2024-09-20T19:34:18.414 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T19:34:18.430 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T19:34:18.523 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912817 branch: wip-adk-testing-2024-09-20-0910 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} email: adking@redhat.com first_in_suite: false job_id: '7912817' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-09-20-0910 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 workunit: branch: wip-adk-testing-2024-09-20-0910 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 8031 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-09-20-0910 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 targets: smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCKuRP7Ssi6ffPRUOv245cO6kiE3ELHhtr9nemPaNV/NRvAntwx89psAdsv+5qCYjPsnKRFfWzxvkT89LdjMqYA= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ7MpPphvVvrd2sY2sA0ogP06i8T21+OEx+n7WJZvN4B0Va8v2g64gxJBjCm3tjupmrBGyCRHayr1QHxY0qj2E= tasks: - cephadm: compiled_cephadm_branch: reef image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_18:25:04 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T19:34:18.523 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa; will attempt to use it 2024-09-20T19:34:18.524 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks 2024-09-20T19:34:18.524 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T19:34:18.525 INFO:teuthology.task.internal:Checking packages... 2024-09-20T19:34:18.546 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7' 2024-09-20T19:34:18.546 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:34:18.546 INFO:teuthology.packaging:ref: None 2024-09-20T19:34:18.546 INFO:teuthology.packaging:tag: None 2024-09-20T19:34:18.546 INFO:teuthology.packaging:branch: wip-adk-testing-2024-09-20-0910 2024-09-20T19:34:18.546 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:34:18.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-09-20-0910 2024-09-20T19:34:18.697 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5127.ge8baca1e 2024-09-20T19:34:18.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T19:34:18.711 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T19:34:18.711 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T19:34:18.732 INFO:teuthology.task.internal:Saving configuration 2024-09-20T19:34:18.744 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T19:34:18.750 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T19:34:18.777 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:29:27.839598', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCKuRP7Ssi6ffPRUOv245cO6kiE3ELHhtr9nemPaNV/NRvAntwx89psAdsv+5qCYjPsnKRFfWzxvkT89LdjMqYA='} 2024-09-20T19:34:18.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:29:27.838111', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ7MpPphvVvrd2sY2sA0ogP06i8T21+OEx+n7WJZvN4B0Va8v2g64gxJBjCm3tjupmrBGyCRHayr1QHxY0qj2E='} 2024-09-20T19:34:18.799 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T19:34:18.806 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T19:34:18.806 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-20T19:34:18.806 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T19:34:18.908 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8b520bc280>, signals=[15]) 2024-09-20T19:34:18.909 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T19:34:18.915 INFO:teuthology.task.internal:Opening connections... 2024-09-20T19:34:18.915 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-09-20T19:34:18.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:34:18.994 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-09-20T19:34:18.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:34:19.072 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T19:34:19.079 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-09-20T19:34:19.097 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-09-20T19:34:19.098 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-09-20T19:34:19.156 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T19:34:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T19:34:19.158 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-09-20T19:34:19.179 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-09-20T19:34:19.198 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-09-20T19:34:19.198 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-09-20T19:34:19.256 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T19:34:19.257 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T19:34:19.257 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-09-20T19:34:19.271 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T19:34:19.280 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T19:34:19.287 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T19:34:19.287 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-09-20T19:34:19.290 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-09-20T19:34:19.313 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T19:34:19.319 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T19:34:19.319 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-09-20T19:34:19.347 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-09-20T19:34:19.443 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T19:34:19.495 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T19:34:19.514 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T19:34:19.514 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T19:34:19.515 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T19:34:19.515 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T19:34:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.532 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-20T19:34:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:34:19.534 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-20T19:34:19.591 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-503.el9.x86_64 2024-09-20T19:34:19.591 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-503.el9.x86_64 2024-09-20T19:34:19.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-09-20T19:34:19.593 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-503.el9.x86_64 2024-09-20T19:34:19.593 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-503.el9.x86_64 2024-09-20T19:34:19.593 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-09-20T19:34:23.144 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-20T19:34:23.373 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-20T19:34:25.174 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-20T19:34:27.128 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 20 MB 00:02 2024-09-20T19:34:29.459 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-20T19:34:31.319 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 19 kB 00:00 2024-09-20T19:34:31.503 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-20T19:34:32.526 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-20T19:34:35.208 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 7.7 kB/s | 19 kB 00:02 2024-09-20T19:34:36.033 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-20T19:34:37.953 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-20T19:34:39.818 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T19:34:39.818 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T19:34:39.915 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-09-20T19:34:39.922 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-20T19:34:39.922 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:39.922 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-20T19:34:39.922 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-09-20T19:34:39.923 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-09-20T19:34:39.924 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-09-20T19:34:39.924 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:34:39.925 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 79 M 2024-09-20T19:34:39.925 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 126 M 2024-09-20T19:34:39.925 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-09-20T19:34:40.021 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 468 kB/s | 36 kB 00:00 2024-09-20T19:34:41.413 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-20T19:34:43.328 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T19:34:43.328 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T19:34:43.362 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.6 MB/s | 16 MB 00:03 2024-09-20T19:34:43.416 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-20T19:34:43.422 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-20T19:34:43.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:34:43.424 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 79 M 2024-09-20T19:34:43.424 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 126 M 2024-09-20T19:34:43.424 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-20T19:34:44.012 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 8.5 MB/s | 35 MB 00:04 2024-09-20T19:34:44.340 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 61 kB/s | 36 kB 00:00 2024-09-20T19:34:44.404 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.5 MB/s | 29 MB 00:04 2024-09-20T19:34:44.404 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:44.404 INFO:teuthology.orchestra.run.smithi184.stdout:Total 18 MB/s | 79 MB 00:04 2024-09-20T19:34:45.096 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-09-20T19:34:45.132 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-20T19:34:45.426 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-09-20T19:34:45.427 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-09-20T19:34:45.823 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-20T19:34:46.399 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-09-20T19:34:46.399 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-09-20T19:34:46.840 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-20T19:34:46.841 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:46.841 INFO:teuthology.orchestra.run.smithi196.stdout:Total 23 MB/s | 79 MB 00:03 2024-09-20T19:34:47.530 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-20T19:34:47.901 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-20T19:34:47.901 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-20T19:34:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-09-20T19:34:48.880 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-20T19:34:48.881 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-20T19:34:49.549 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:34:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:34:51.347 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-20T19:34:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:34:52.125 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:34:52.377 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:34:52.518 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:34:54.922 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:34:55.077 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:34:57.332 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:34:57.435 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:34:59.904 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:35:00.027 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:02.264 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:04.877 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:40.628 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:40.676 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:35:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:35:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:35:41.952 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:35:41.953 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-09-20T19:35:42.315 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:35:42.397 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:42.398 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:35:42.398 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T19:35:42.398 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T19:35:42.398 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T19:35:42.398 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T19:35:42.398 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T19:35:42.398 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:35:42.456 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:42.457 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:35:42.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-09-20T19:35:42.489 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-09-20T19:35:42.489 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 82.el9 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8274670 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-09-20T19:35:42.490 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:35:42.491 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T19:35:42.492 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T19:35:42.492 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-508.el9.x86_64 2024-09-20T19:35:42.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T19:35:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:42.974 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:43.159 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-09-20T19:35:43.249 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:43.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:35:43.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:35:43.249 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:35:44.076 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:35:44.077 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-20T19:35:44.426 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:35:44.454 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T19:35:44.483 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-09-20T19:35:44.485 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-20T19:35:44.501 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.WXes7HYL6I 2024-09-20T19:35:44.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WXes7HYL6I 2024-09-20T19:35:44.520 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.520 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:35:44.520 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T19:35:44.520 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T19:35:44.521 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T19:35:44.521 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T19:35:44.521 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T19:35:44.521 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:35:44.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.WXes7HYL6I 2024-09-20T19:35:44.579 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.580 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:35:44.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-09-20T19:35:44.612 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 82.el9 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8274670 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-09-20T19:35:44.613 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:35:44.614 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T19:35:44.615 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T19:35:44.615 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-508.el9.x86_64 2024-09-20T19:35:44.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T19:35:44.711 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.WXes7HYL6I is 6KB 2024-09-20T19:35:44.760 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.WXes7HYL6I 2024-09-20T19:35:44.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T19:35:44.837 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T19:35:44.838 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T19:35:44.838 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T19:35:44.838 INFO:teuthology.orchestra.run.smithi184.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T19:35:44.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T19:35:44.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-09-20T19:35:45.278 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-09-20T19:35:46.639 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T19:35:46.655 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-09-20T19:35:46.667 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-09-20T19:35:46.682 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.E2gE3Bhi6w 2024-09-20T19:35:46.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.E2gE3Bhi6w 2024-09-20T19:35:46.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.E2gE3Bhi6w 2024-09-20T19:35:46.891 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.E2gE3Bhi6w is 6KB 2024-09-20T19:35:46.939 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.E2gE3Bhi6w 2024-09-20T19:35:46.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T19:35:47.021 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T19:35:47.021 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T19:35:47.021 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T19:35:47.021 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T19:35:47.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T19:35:47.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-09-20T19:36:15.022 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T19:36:15.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-20T19:36:15.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:36:17.162 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T19:36:17.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-20T19:36:17.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:37:15.083 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T19:37:17.166 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T19:37:24.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-20T19:37:24.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:37:24.501 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-20T19:37:24.914 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-20T19:37:24.914 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T19:37:24.915 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-20T19:37:24.930 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-508.el9.x86_64 2024-09-20T19:37:24.930 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T19:37:24.930 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T19:37:24.930 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T19:37:25.931 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T19:37:25.932 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T19:37:26.002 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-09-20T19:37:26.024 DEBUG:teuthology.parallel:result is None 2024-09-20T19:37:26.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-20T19:37:26.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:37:26.550 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-20T19:37:26.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-20T19:37:26.950 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T19:37:26.950 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-20T19:37:26.965 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-508.el9.x86_64 2024-09-20T19:37:26.965 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T19:37:26.965 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T19:37:26.965 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T19:37:27.966 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T19:37:27.966 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T19:37:28.076 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-09-20T19:37:28.116 DEBUG:teuthology.parallel:result is None 2024-09-20T19:37:28.116 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T19:37:28.123 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T19:37:28.123 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T19:37:28.126 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T19:37:28.141 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T19:37:28.149 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T19:37:28.155 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T19:37:28.155 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T19:37:28.182 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T19:37:28.254 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T19:37:28.260 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T19:37:28.260 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T19:37:28.263 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T19:37:28.290 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:37:28.300 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:37:28.544 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:37:28.555 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:37:28.557 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T19:37:28.566 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T19:37:28.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T19:37:28.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T19:37:28.594 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T19:37:28.602 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T19:37:28.602 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T19:37:28.635 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T19:37:28.654 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:37:28.745 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:37:28.804 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:37:28.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:37:28.987 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:37:28.987 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T19:37:29.049 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:37:29.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:37:29.135 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:37:29.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:37:29.269 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:37:29.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T19:37:29.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-09-20T19:37:29.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-09-20T19:37:29.424 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:37:29.443 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:37:29.916 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T19:37:29.930 INFO:teuthology.task.internal:Starting timer... 2024-09-20T19:37:29.930 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T19:37:29.966 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T19:37:29.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-09-20T19:37:30.227 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-09-20T19:37:30.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-09-20T19:37:30.285 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-09-20T19:37:30.287 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T19:37:30.287 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-09-20T19:37:30.394 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-09-20T19:37:30.394 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-09-20T19:37:30.429 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-09-20T19:37:30.429 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi184.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-09-20T19:37:30.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T19:37:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T19:37:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:37:30.499 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T19:37:30.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-09-20T19:37:30.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-09-20T19:37:30.587 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T19:37:30.596 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T19:37:30.606 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T19:37:30.607 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi184.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T19:41:12.904 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi184.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-09-20T19:41:12.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-20T19:41:12.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:41:12.982 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-20T19:41:13.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-20T19:41:13.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-20T19:41:13.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:41:13.140 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-20T19:41:13.214 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-20T19:41:13.214 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T19:41:13.229 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T19:41:13.230 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:41:13.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:41:13.234 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:41:13.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:41:13.259 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T19:41:13.273 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T19:41:13.283 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T19:41:13.297 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-09-20T19:41:13.298 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T19:41:13.308 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-09-20T19:41:13.322 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T19:41:13.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-09-20T19:41:13.333 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-09-20T19:41:13.336 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T19:41:13.347 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T19:41:13.361 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T19:41:13.388 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:41:13.391 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:41:13.391 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-09-20T19:41:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.392 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.419 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.423 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:41:13.424 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T19:41:13.518 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'} 2024-09-20T19:41:13.518 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T19:41:13.519 INFO:tasks.cephadm:Cluster fsid is 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:41:13.519 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T19:41:13.519 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.184', 'mon.c': '[v2:172.21.15.184:3301,v1:172.21.15.184:6790]', 'mon.b': '172.21.15.196'} 2024-09-20T19:41:13.519 INFO:tasks.cephadm:First mon is mon.a on smithi184 2024-09-20T19:41:13.519 INFO:tasks.cephadm:First mgr is y 2024-09-20T19:41:13.519 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T19:41:13.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-09-20T19:41:13.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-09-20T19:41:13.572 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-20T19:41:13.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:41:13.714 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83188/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5127-ge8baca1e', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5127.ge8baca1e'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 14:53:51.724057', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-09-20-0910', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-20T19:41:13.947 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-20T19:41:13.948 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T19:41:13.948 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T19:41:13.948 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:14.462 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 19:41 /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:14.462 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:14.970 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 19:41 /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:14.971 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:14.988 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:41:15.024 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-20T19:41:15.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T19:41:15.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T19:41:16.582 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T19:41:16.588 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-20T19:42:08.347 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-20T19:42:09.772 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:42:09.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T19:42:09.773 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T19:42:09.773 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-09-20T19:42:09.773 INFO:teuthology.orchestra.run.smithi184.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T19:42:09.773 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:42:09.773 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:42:09.790 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-09-20T19:42:09.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-09-20T19:42:09.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-09-20T19:42:09.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-09-20T19:42:09.940 INFO:tasks.cephadm:Writing seed config... 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T19:42:09.941 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T19:42:09.942 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:42:09.942 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T19:42:09.957 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4b598dd2-7788-11ef-baf5-efdc52797490 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T19:42:09.957 DEBUG:teuthology.orchestra.run.smithi184:mon.a> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T19:42:09.999 DEBUG:teuthology.orchestra.run.smithi184:mgr.y> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T19:42:10.042 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T19:42:10.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.184 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:42:10.194 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:42:10.194 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '4b598dd2-7788-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.184', '--skip-admin-label'] 2024-09-20T19:42:10.218 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 5.2.2 2024-09-20T19:42:10.219 INFO:teuthology.orchestra.run.smithi184.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T19:42:10.219 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying podman|docker is present... 2024-09-20T19:42:10.249 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 5.2.2 2024-09-20T19:42:10.249 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying lvm2 is present... 2024-09-20T19:42:10.249 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying time synchronization is in place... 2024-09-20T19:42:10.256 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:42:10.256 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:42:10.262 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:42:10.263 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:42:10.269 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout enabled 2024-09-20T19:42:10.275 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout active 2024-09-20T19:42:10.275 INFO:teuthology.orchestra.run.smithi184.stdout:Unit chronyd.service is enabled and running 2024-09-20T19:42:10.275 INFO:teuthology.orchestra.run.smithi184.stdout:Repeating the final host check... 2024-09-20T19:42:10.297 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 5.2.2 2024-09-20T19:42:10.297 INFO:teuthology.orchestra.run.smithi184.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-20T19:42:10.297 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl is present 2024-09-20T19:42:10.297 INFO:teuthology.orchestra.run.smithi184.stdout:lvcreate is present 2024-09-20T19:42:10.304 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:42:10.304 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:42:10.310 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:42:10.310 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:42:10.316 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout enabled 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout active 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Unit chronyd.service is enabled and running 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Host looks OK 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Cluster fsid: 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Acquiring lock 140242007400608 on /run/cephadm/4b598dd2-7788-11ef-baf5-efdc52797490.lock 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Lock 140242007400608 acquired on /run/cephadm/4b598dd2-7788-11ef-baf5-efdc52797490.lock 2024-09-20T19:42:10.323 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying IP 172.21.15.184 port 3300 ... 2024-09-20T19:42:10.324 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying IP 172.21.15.184 port 6789 ... 2024-09-20T19:42:10.324 INFO:teuthology.orchestra.run.smithi184.stdout:Base mon IP(s) is [172.21.15.184:3300, 172.21.15.184:6789], mon addrv is [v2:172.21.15.184:3300,v1:172.21.15.184:6789] 2024-09-20T19:42:10.327 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.184 metric 100 2024-09-20T19:42:10.327 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.184 metric 100 2024-09-20T19:42:10.330 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T19:42:10.330 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T19:42:10.330 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link noprefixroute 2024-09-20T19:42:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:42:10.333 INFO:teuthology.orchestra.run.smithi184.stdout:Mon IP `172.21.15.184` is in CIDR network `172.21.0.0/20` 2024-09-20T19:42:10.333 INFO:teuthology.orchestra.run.smithi184.stdout:Mon IP `172.21.15.184` is in CIDR network `172.21.0.0/20` 2024-09-20T19:42:10.333 INFO:teuthology.orchestra.run.smithi184.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-20T19:42:10.333 INFO:teuthology.orchestra.run.smithi184.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T19:42:10.334 INFO:teuthology.orchestra.run.smithi184.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Getting image source signatures 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying blob sha256:1f571018c56d8512357839d2ed445827ec316c3aed0fbfc68aff9046d4aceb99 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying config sha256:4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T19:42:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-20T19:42:11.938 INFO:teuthology.orchestra.run.smithi184.stdout:ceph: stdout ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T19:42:11.939 INFO:teuthology.orchestra.run.smithi184.stdout:Ceph version: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T19:42:11.939 INFO:teuthology.orchestra.run.smithi184.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T19:42:12.744 INFO:teuthology.orchestra.run.smithi184.stdout:stat: stdout 167 167 2024-09-20T19:42:12.744 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial keys... 2024-09-20T19:42:13.684 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQAV0O1mmc8eBBAARwQHfLRZMU97qofLSdXTxQ== 2024-09-20T19:42:14.547 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQAW0O1mkHr0ABAA4TSf1hD1zN6UoK/V9qi8sw== 2024-09-20T19:42:15.494 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQAW0O1m+UPqMxAA8flI6fRhY1SRay3p34daNw== 2024-09-20T19:42:15.495 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial monmap... 2024-09-20T19:42:16.416 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool for a [v2:172.21.15.184:3300,v1:172.21.15.184:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:setting min_mon_release = pacific 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: set fsid to 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:42:16.417 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:42:16.418 INFO:teuthology.orchestra.run.smithi184.stdout:Creating mon... 2024-09-20T19:42:17.238 INFO:teuthology.orchestra.run.smithi184.stdout:create mon.a on 2024-09-20T19:42:17.598 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T19:42:17.774 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target. 2024-09-20T19:42:17.774 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target. 2024-09-20T19:42:18.033 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a 2024-09-20T19:42:18.033 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T19:42:18.207 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target.wants/ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490@.service. 2024-09-20T19:42:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 podman[23306]: 2024-09-20 19:42:18.352578979 +0000 UTC m=+0.016005658 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T19:42:18.738 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:42:18.739 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-09-20T19:42:18.745 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:42:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:42:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-09-20T19:42:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:42:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mon to start... 2024-09-20T19:42:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mon... 2024-09-20T19:42:18.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 podman[23306]: 2024-09-20 19:42:18.640635684 +0000 UTC m=+0.304062364 container init 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T19:42:18.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 podman[23306]: 2024-09-20 19:42:18.643853612 +0000 UTC m=+0.307280278 container start 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:42:18.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 ceph-mon[23325]: mkfs 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:18.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 ceph-mon[23325]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:42:18.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 bash[23306]: 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 2024-09-20T19:42:18.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:18 smithi184 systemd[1]: Started Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout id: 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout services: 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.566325s) 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:19.827 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout data: 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon is available 2024-09-20T19:42:19.828 INFO:teuthology.orchestra.run.smithi184.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T19:42:19.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:42:19.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: monmap e1: 1 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:42:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: fsmap 2024-09-20T19:42:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:42:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: mgrmap e1: no daemons active 2024-09-20T19:42:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:19 smithi184 ceph-mon[23325]: from='client.? 172.21.15.184:0/2772903460' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T19:42:20.848 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout fsid = 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.184:3300,v1:172.21.15.184:6789] 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:20.849 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:42:20.850 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:42:20.850 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:42:20.850 INFO:teuthology.orchestra.run.smithi184.stdout:Generating new minimal ceph.conf... 2024-09-20T19:42:20.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:20 smithi184 ceph-mon[23325]: from='client.? 172.21.15.184:0/2095298319' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T19:42:20.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:20 smithi184 ceph-mon[23325]: from='client.? 172.21.15.184:0/2095298319' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T19:42:21.786 INFO:teuthology.orchestra.run.smithi184.stdout:Restarting the monitor... 2024-09-20T19:42:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:21 smithi184 ceph-mon[23325]: from='client.? 172.21.15.184:0/3318761289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:21 smithi184 systemd[1]: Stopping Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:42:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[23321]: 2024-09-20T19:42:22.079+0000 7f4f56cbd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:42:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[23321]: 2024-09-20T19:42:22.079+0000 7f4f56cbd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T19:42:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 podman[23569]: 2024-09-20 19:42:22.285169429 +0000 UTC m=+0.318571886 container died 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-20T19:42:22.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 podman[23569]: 2024-09-20 19:42:22.652415173 +0000 UTC m=+0.685817620 container cleanup 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-20T19:42:22.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 bash[23569]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:42:23.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:22 smithi184 podman[23581]: 2024-09-20 19:42:22.911872938 +0000 UTC m=+0.621505697 container remove 4c44f2931bea13d45bf70f47f5c7f5fa30a992c6bee5ae94a92040669d5021f1 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS) 2024-09-20T19:42:23.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T19:42:23.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 systemd[1]: Stopped Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:42:23.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 systemd[1]: Starting Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:42:23.486 INFO:teuthology.orchestra.run.smithi184.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 podman[23673]: 2024-09-20 19:42:23.254240236 +0000 UTC m=+0.090907267 container create 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 podman[23673]: 2024-09-20 19:42:23.179126822 +0000 UTC m=+0.015793866 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 podman[23673]: 2024-09-20 19:42:23.423487531 +0000 UTC m=+0.260154562 container init 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 podman[23673]: 2024-09-20 19:42:23.426540469 +0000 UTC m=+0.263207505 container start 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T19:42:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: pidfile_write: ignore empty --pid-file 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: load: jerasure load: lrc 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Git sha 0 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: DB SUMMARY 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: DB Session ID: U9C13P01FLEF2X3PXH4U 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83676 ; 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:42:23.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:42:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:42:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.env: 0x560c9e3e3bc0 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.info_log: 0x560c9f470be0 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.statistics: (nil) 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.use_fsync: 0 2024-09-20T19:42:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.db_log_dir: 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.wal_dir: 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:42:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.write_buffer_manager: 0x560c9f47e320 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:42:23.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.unordered_write: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.row_cache: None 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.wal_filter: None 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.wal_compression: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:42:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:42:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_open_files: -1 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Compression algorithms supported: 2024-09-20T19:42:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kZSTD supported: 0 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.merge_operator: 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_filter: None 2024-09-20T19:42:23.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560c9f4707e0) 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_type: 0 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_index_type: 0 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_shortening: 1 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: checksum: 4 2024-09-20T19:42:23.525 INFO:journalctl@ceph.mon.a.smithi184.stdout: no_block_cache: 0 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache: 0x560c9f465350 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_options: 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: capacity : 536870912 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_shard_bits : 4 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: strict_capacity_limit : 0 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_compressed: (nil) 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: persistent_cache: (nil) 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size: 4096 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size_deviation: 10 2024-09-20T19:42:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_restart_interval: 16 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_block_restart_interval: 1 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: metadata_block_size: 4096 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: partition_filters: 0 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: use_delta_encoding: 1 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: filter_policy: bloomfilter 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: whole_key_filtering: 1 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: verify_compression: 0 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: format_version: 5 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_index_compression: 1 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_align: 0 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: prepopulate_block_cache: 0 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:42:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression: NoCompression 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.num_levels: 7 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:42:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:42:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:42:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:42:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.ttl: 2592000 2024-09-20T19:42:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T19:42:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861343453087, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861343453917, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80325, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 78486, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 72797, "raw_average_value_size": 345, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726861343, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3", "db_session_id": "U9C13P01FLEF2X3PXH4U", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861343454010, "job": 1, "event": "recovery_finished"} 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560c9f490e00 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: DB pointer 0x560c9f584000 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** DB Stats ** 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:42:23.534 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: L0 2/0 80.30 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Sum 2/0 80.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.535 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative compaction: 0.00 GB write, 26.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval compaction: 0.00 GB write, 26.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:42:23.536 INFO:journalctl@ceph.mon.a.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache BinnedLRUCache@0x560c9f465350#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: starting mon.a rank 0 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???) e1 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).mds e1 new map 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).mds e1 print_map 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: e1 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: legacy client fscid: -1 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:42:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout: No filesystems configured 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 bash[23673]: 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 systemd[1]: Started Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: monmap e1: 1 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: fsmap 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:42:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:23 smithi184 ceph-mon[23692]: mgrmap e1: no daemons active 2024-09-20T19:42:24.592 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T19:42:24.613 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:42:24.613 INFO:teuthology.orchestra.run.smithi184.stdout:Creating mgr... 2024-09-20T19:42:24.614 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T19:42:24.614 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T19:42:24.866 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y 2024-09-20T19:42:24.866 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T19:42:25.012 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490.target.wants/ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-4b598dd2-7788-11ef-baf5-efdc52797490@.service. 2024-09-20T19:42:25.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:24 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3170565147' entity='client.admin' 2024-09-20T19:42:25.233 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 systemd[1]: Starting Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:42:25.535 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 podman[23905]: 2024-09-20 19:42:25.231770263 +0000 UTC m=+0.105115298 container create c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d) 2024-09-20T19:42:25.535 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 podman[23905]: 2024-09-20 19:42:25.142506327 +0000 UTC m=+0.015851327 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T19:42:25.535 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 podman[23905]: 2024-09-20 19:42:25.43378338 +0000 UTC m=+0.307128382 container init c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, RELEASE=reef-0b2cc4d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:42:25.535 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 podman[23905]: 2024-09-20 19:42:25.436348594 +0000 UTC m=+0.309693587 container start c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-20T19:42:25.535 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 bash[23905]: c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 2024-09-20T19:42:25.549 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:42:25.549 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-09-20T19:42:25.556 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:42:25.556 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:42:25.556 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-09-20T19:42:25.556 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:42:25.564 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:42:25.565 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr to start... 2024-09-20T19:42:25.572 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr... 2024-09-20T19:42:25.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 systemd[1]: Started Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:42:25.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:25.553+0000 7f60e3a5e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:42:25.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:25.618+0000 7f60e3a5e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:42:25.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:25.768+0000 7f60e3a5e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:42:26.043 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:25.837+0000 7f60e3a5e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:42:26.043 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:25.988+0000 7f60e3a5e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:42:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:26 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2901920147' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T19:42:26.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:26.134+0000 7f60e3a5e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "4b598dd2-7788-11ef-baf5-efdc52797490", 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:42:26.563 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:42:26.564 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:42:26.565 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:42:26.565 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:42:26.568 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:42:26.569 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:42:18.687843+0000", 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:42:26.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr not available, waiting (1/15)... 2024-09-20T19:42:26.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:26.446+0000 7f60e3a5e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:42:26.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:26.508+0000 7f60e3a5e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:42:26.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:26.692+0000 7f60e3a5e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:42:28.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:27.548+0000 7f60e3a5e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:42:28.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:27.613+0000 7f60e3a5e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:42:28.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.319+0000 7f60e3a5e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:42:28.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.431+0000 7f60e3a5e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:42:28.860 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.572+0000 7f60e3a5e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:42:28.860 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.643+0000 7f60e3a5e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:42:28.860 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.800+0000 7f60e3a5e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:42:29.117 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.885+0000 7f60e3a5e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:42:29.117 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:28.948+0000 7f60e3a5e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:42:29.391 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1567872466' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T19:42:29.392 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.171+0000 7f60e3a5e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:42:29.392 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.236+0000 7f60e3a5e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:42:29.392 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.303+0000 7f60e3a5e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:42:29.688 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.389+0000 7f60e3a5e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:42:29.688 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.447+0000 7f60e3a5e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:42:29.689 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:29.507+0000 7f60e3a5e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:42:29.696 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:29.696 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:42:29.696 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "4b598dd2-7788-11ef-baf5-efdc52797490", 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-20T19:42:29.697 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.698 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.699 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:42:18.687843+0000", 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:29.700 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:29.701 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:42:29.701 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:42:29.701 INFO:teuthology.orchestra.run.smithi184.stdout:mgr not available, waiting (2/15)... 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:30.288+0000 7f60e3a5e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: Activating manager daemon y 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: mgrmap e2: y(active, starting, since 0.00335406s) 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:42:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: Manager daemon y is now available 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' 2024-09-20T19:42:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:30 smithi184 ceph-mon[23692]: from='mgr.14100 172.21.15.184:0/3750083648' entity='mgr.y' 2024-09-20T19:42:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:32 smithi184 ceph-mon[23692]: mgrmap e3: y(active, since 1.00666s) 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "4b598dd2-7788-11ef-baf5-efdc52797490", 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:42:32.907 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:42:32.908 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:42:32.909 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:42:32.912 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:42:32.913 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:42:18.687843+0000", 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:42:32.914 INFO:teuthology.orchestra.run.smithi184.stdout:mgr is available 2024-09-20T19:42:33.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:33 smithi184 ceph-mon[23692]: mgrmap e4: y(active, since 2s) 2024-09-20T19:42:33.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:33 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1362918187' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T19:42:34.171 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:34.171 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout fsid = 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:42:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling cephadm module... 2024-09-20T19:42:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3550910807' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T19:42:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3550910807' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T19:42:38.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:37 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2934548283' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T19:42:38.045 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setuser ceph since I am not root 2024-09-20T19:42:38.046 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setgroup ceph since I am not root 2024-09-20T19:42:38.046 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:37.841+0000 7f175f5ab040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:42:38.046 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:37.898+0000 7f175f5ab040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:42:38.046 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.044+0000 7f175f5ab040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:42:38.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.110+0000 7f175f5ab040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:42:38.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.254+0000 7f175f5ab040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:42:38.700 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.394+0000 7f175f5ab040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:42:38.700 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.699+0000 7f175f5ab040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:42:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2934548283' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T19:42:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-mon[23692]: mgrmap e5: y(active, since 7s) 2024-09-20T19:42:39.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.760+0000 7f175f5ab040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:42:39.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:38.943+0000 7f175f5ab040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:42:40.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:39.818+0000 7f175f5ab040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:42:40.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:39.885+0000 7f175f5ab040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:42:40.893 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:40.625+0000 7f175f5ab040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:42:40.893 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:40.745+0000 7f175f5ab040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:42:41.178 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:40.892+0000 7f175f5ab040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:42:41.179 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:40.955+0000 7f175f5ab040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:42:41.179 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.106+0000 7f175f5ab040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:42:41.179 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.176+0000 7f175f5ab040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:42:41.463 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.238+0000 7f175f5ab040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:42:41.463 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.397+0000 7f175f5ab040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:42:41.463 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.461+0000 7f175f5ab040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:42:41.730 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.528+0000 7f175f5ab040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:42:41.731 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.611+0000 7f175f5ab040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:42:41.731 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.670+0000 7f175f5ab040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:42:41.731 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:41.729+0000 7f175f5ab040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:42:42.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:42.509+0000 7f175f5ab040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:42:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: Active manager daemon y restarted 2024-09-20T19:42:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: Activating manager daemon y 2024-09-20T19:42:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T19:42:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: mgrmap e6: y(active, starting, since 0.00389084s) 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: Manager daemon y is now available 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:42:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:42 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:42:44.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:43 smithi184 ceph-mon[23692]: Found migration_current of "None". Setting to last migration. 2024-09-20T19:42:44.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:43 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:44.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:43 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:44.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:43 smithi184 ceph-mon[23692]: mgrmap e7: y(active, since 1.00578s) 2024-09-20T19:42:44.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:44.036+0000 7f16c728b640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:42:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:42:44.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Traceback (most recent call last): 2024-09-20T19:42:44.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._connections.run(self.expiration_interval) 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._run(expiration_interval) 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s = self.context.wrap_socket( 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: return self.sslsocket_class._create( 2024-09-20T19:42:44.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:42:44.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self.do_handshake() 2024-09-20T19:42:44.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:42:44.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._sslobj.do_handshake() 2024-09-20T19:42:44.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:42:44.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:42:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:44 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: [20/Sep/2024:19:42:43] ENGINE Bus STARTING 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: [20/Sep/2024:19:42:43] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: [20/Sep/2024:19:42:44] ENGINE Serving on https://172.21.15.184:7150 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: [20/Sep/2024:19:42:44] ENGINE Bus STARTED 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: [20/Sep/2024:19:42:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:42:46.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:42:46.027 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:42:46.028 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:42:46.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: mgrmap e8: y(active, since 2s) 2024-09-20T19:42:46.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:45 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3430365605' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T19:42:46.029 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:42:46.029 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 8, 2024-09-20T19:42:46.029 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:42:46.029 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T19:42:46.030 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:42:46.030 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:42:46.030 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for the mgr to restart... 2024-09-20T19:42:46.030 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr epoch 8... 2024-09-20T19:42:47.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:46 smithi184 ceph-mon[23692]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:42:47.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:46 smithi184 ceph-mon[23692]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:42:47.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:42:47.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-20T19:42:47.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:42:47.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:42:47.529 INFO:teuthology.orchestra.run.smithi184.stdout:mgr epoch 8 is available 2024-09-20T19:42:47.529 INFO:teuthology.orchestra.run.smithi184.stdout:Setting orchestrator backend to cephadm... 2024-09-20T19:42:49.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:49 smithi184 ceph-mon[23692]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:49.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:49 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:49.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:49 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:49.739 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T19:42:49.740 INFO:teuthology.orchestra.run.smithi184.stdout:Generating ssh key... 2024-09-20T19:42:50.463 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-mon[23692]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Generating public/private rsa key pair. 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Your identification has been saved in /tmp/tmp2_ouerjz/key 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Your public key has been saved in /tmp/tmp2_ouerjz/key.pub 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: The key fingerprint is: 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: SHA256:pdKJUdghq2fuS+b0vLlhyXEZEvLGS8HTvFo8oaym6/I ceph-4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: The key's randomart image is: 2024-09-20T19:42:50.738 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: +---[RSA 3072]----+ 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | o+=+ | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | .B+o+ | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | o.*+oo | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | . *o*=o | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | . =.Soo. | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | +oo.+ | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | o= = | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | . .* + o | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: | +E.+.*o | 2024-09-20T19:42:50.739 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: +----[SHA256]-----+ 2024-09-20T19:42:51.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:51 smithi184 ceph-mon[23692]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:51.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:51 smithi184 ceph-mon[23692]: Generating ssh key... 2024-09-20T19:42:51.570 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:51 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:51.570 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:51 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:52.126 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcEMzNqK2HG7a70Uh6k8fWeYPIXoQsdPC6r3TMNdB6mcuHk2pKfUGbvCgFZa5/UA6PFfkUWV04eyZEyeqa3MrTDdSIjjMdrjp4CBgHV0PYCQYbAD0p0qUpijkgtKn9cRnQ70W6bNdHJYNoA907C9PLZ0/J9vOdiCDsQpGOiCRmx6Nmp3YV2fiugkFPexHBJ7i9WpUwfiMVcLSxKpOthpMQES5+340o3cBBYawBRX13x5Aurldp2dD9K2FBHfnhlJTxOI+7KT8EQvV3OXzoyyoUeTlTf7qBuiNNKt+LkmESgyiNJX7nDYbvaphyY3k7L4lijRNLekfioRXzkFaAadTnezDiQ1qEtGJBIlsMHro02y2LueZiOF9i33HBvjX0dXp2J9ALIHEhCG/ZyII3IrgNHRzjz4CIcHLVQIe2mteLUH2MoNdTiq3Cg15PuQ4ZJLQnspMjrHCLYKBxJe9OD/jqe9IH66q6Uo8Zw5PJzFVWNZc1yVjsWBFH+BoP6u42Cas= ceph-4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:42:52.127 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:42:52.127 INFO:teuthology.orchestra.run.smithi184.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T19:42:52.136 INFO:teuthology.orchestra.run.smithi184.stdout:Adding host smithi184... 2024-09-20T19:42:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:52 smithi184 ceph-mon[23692]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:53.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:53 smithi184 ceph-mon[23692]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "addr": "172.21.15.184", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:54.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:54 smithi184 ceph-mon[23692]: Deploying cephadm binary to smithi184 2024-09-20T19:42:55.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Added host 'smithi184' with addr '172.21.15.184' 2024-09-20T19:42:55.859 INFO:teuthology.orchestra.run.smithi184.stdout:Deploying unmanaged mon service... 2024-09-20T19:42:56.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:56 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:56.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:56 smithi184 ceph-mon[23692]: Added host smithi184 2024-09-20T19:42:56.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:56 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:57.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:57 smithi184 ceph-mon[23692]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:57.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:57 smithi184 ceph-mon[23692]: Saving service mon spec with placement count:5 2024-09-20T19:42:57.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:57 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:57.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:57 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:57.770 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T19:42:57.770 INFO:teuthology.orchestra.run.smithi184.stdout:Deploying unmanaged mgr service... 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: Saving service mgr spec with placement count:2 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:42:59 smithi184 ceph-mon[23692]: from='mgr.14118 172.21.15.184:0/4200050892' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]': finished 2024-09-20T19:42:59.514 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T19:43:01.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:01 smithi184 ceph-mon[23692]: Deploying daemon agent.smithi184 on smithi184 2024-09-20T19:43:01.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:01 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4177105091' entity='client.admin' 2024-09-20T19:43:01.792 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling the dashboard module... 2024-09-20T19:43:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3935425959' entity='client.admin' 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3289642476' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-mon[23692]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setuser ceph since I am not root 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setgroup ceph since I am not root 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.351+0000 7f8fc167e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:43:03.530 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.409+0000 7f8fc167e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:43:03.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.562+0000 7f8fc167e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:43:03.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.627+0000 7f8fc167e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:43:03.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.776+0000 7f8fc167e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:43:04.124 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:03.920+0000 7f8fc167e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:43:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3289642476' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T19:43:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:04 smithi184 ceph-mon[23692]: mgrmap e9: y(active, since 20s) 2024-09-20T19:43:04.423 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:04.236+0000 7f8fc167e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:43:04.423 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:04.296+0000 7f8fc167e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:43:04.719 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:04.484+0000 7f8fc167e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:43:05.014 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for the mgr to restart... 2024-09-20T19:43:05.015 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr epoch 9... 2024-09-20T19:43:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:05 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2142962483' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T19:43:05.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:05.374+0000 7f8fc167e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:43:05.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:05.443+0000 7f8fc167e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:43:06.405 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.151+0000 7f8fc167e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:43:06.405 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.265+0000 7f8fc167e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:43:06.680 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.403+0000 7f8fc167e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:43:06.680 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.464+0000 7f8fc167e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:43:06.680 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.610+0000 7f8fc167e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:43:06.680 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.678+0000 7f8fc167e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:43:06.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.736+0000 7f8fc167e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:43:06.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.893+0000 7f8fc167e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:43:06.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:06.958+0000 7f8fc167e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:43:07.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:07.023+0000 7f8fc167e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:43:07.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:07.106+0000 7f8fc167e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:43:07.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:07.164+0000 7f8fc167e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:43:07.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:07.223+0000 7f8fc167e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:07.999+0000 7f8fc167e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: Active manager daemon y restarted 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: Activating manager daemon y 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: mgrmap e10: y(active, starting, since 0.0031683s) 2024-09-20T19:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: Manager daemon y is now available 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:08 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:43:09.740 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:43:09.740 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-20T19:43:09.740 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:43:09.741 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:43:09.741 INFO:teuthology.orchestra.run.smithi184.stdout:mgr epoch 9 is available 2024-09-20T19:43:09.741 INFO:teuthology.orchestra.run.smithi184.stdout:Generating a dashboard self-signed certificate... 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:09.857+0000 7f8f27b08640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:43:09] ENGINE Error in HTTPServer.serve 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Traceback (most recent call last): 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._connections.run(self.expiration_interval) 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._run(expiration_interval) 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:43:09.991 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s = self.context.wrap_socket( 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: return self.sslsocket_class._create( 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self.do_handshake() 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._sslobj.do_handshake() 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:43:09.992 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: mgrmap e11: y(active, since 1.00555s) 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:43:11.007 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-20T19:43:11.007 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial admin user... 2024-09-20T19:43:11.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: [20/Sep/2024:19:43:09] ENGINE Bus STARTING 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: [20/Sep/2024:19:43:09] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: [20/Sep/2024:19:43:09] ENGINE Serving on https://172.21.15.184:7150 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: [20/Sep/2024:19:43:09] ENGINE Bus STARTED 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: [20/Sep/2024:19:43:09] ENGINE Error in HTTPServer.serve 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:43:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:11.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:11 smithi184 ceph-mon[23692]: mgrmap e12: y(active, since 2s) 2024-09-20T19:43:12.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:12 smithi184 ceph-mon[23692]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:12.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:12 smithi184 ceph-mon[23692]: Deploying daemon agent.smithi184 on smithi184 2024-09-20T19:43:12.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:12.396 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$wO/p8v6rqUXU9siOBs4hBu1Ziph0zxcO9YenHphrPT9NTtL.8mmfm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726861391, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T19:43:12.397 INFO:teuthology.orchestra.run.smithi184.stdout:Fetching dashboard port number... 2024-09-20T19:43:13.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:13 smithi184 ceph-mon[23692]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:13.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:13 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1278275574' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T19:43:13.561 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T19:43:13.572 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:43:13.572 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-09-20T19:43:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:43:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-09-20T19:43:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-09-20T19:43:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout:Ceph Dashboard is now available at: 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout: URL: https://smithi184.front.sepia.ceph.com:8443/ 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout: User: admin 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout: Password: ov6p17zkfk 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:13.582 INFO:teuthology.orchestra.run.smithi184.stdout:Saving cluster configuration to /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config directory 2024-09-20T19:43:13.583 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling autotune for osd_memory_target 2024-09-20T19:43:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:15 smithi184 ceph-mon[23692]: mgrmap e13: y(active, since 6s) 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: ceph telemetry on 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:For more information see: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Bootstrap complete. 2024-09-20T19:43:15.953 INFO:tasks.cephadm:Fetching config... 2024-09-20T19:43:15.953 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:43:15.953 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T19:43:15.968 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T19:43:15.969 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:43:15.969 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T19:43:16.027 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T19:43:16.027 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:43:16.027 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T19:43:16.149 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T19:43:16.150 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:43:16.150 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T19:43:16.212 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T19:43:16.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcEMzNqK2HG7a70Uh6k8fWeYPIXoQsdPC6r3TMNdB6mcuHk2pKfUGbvCgFZa5/UA6PFfkUWV04eyZEyeqa3MrTDdSIjjMdrjp4CBgHV0PYCQYbAD0p0qUpijkgtKn9cRnQ70W6bNdHJYNoA907C9PLZ0/J9vOdiCDsQpGOiCRmx6Nmp3YV2fiugkFPexHBJ7i9WpUwfiMVcLSxKpOthpMQES5+340o3cBBYawBRX13x5Aurldp2dD9K2FBHfnhlJTxOI+7KT8EQvV3OXzoyyoUeTlTf7qBuiNNKt+LkmESgyiNJX7nDYbvaphyY3k7L4lijRNLekfioRXzkFaAadTnezDiQ1qEtGJBIlsMHro02y2LueZiOF9i33HBvjX0dXp2J9ALIHEhCG/ZyII3IrgNHRzjz4CIcHLVQIe2mteLUH2MoNdTiq3Cg15PuQ4ZJLQnspMjrHCLYKBxJe9OD/jqe9IH66q6Uo8Zw5PJzFVWNZc1yVjsWBFH+BoP6u42Cas= ceph-4b598dd2-7788-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:43:16.345 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcEMzNqK2HG7a70Uh6k8fWeYPIXoQsdPC6r3TMNdB6mcuHk2pKfUGbvCgFZa5/UA6PFfkUWV04eyZEyeqa3MrTDdSIjjMdrjp4CBgHV0PYCQYbAD0p0qUpijkgtKn9cRnQ70W6bNdHJYNoA907C9PLZ0/J9vOdiCDsQpGOiCRmx6Nmp3YV2fiugkFPexHBJ7i9WpUwfiMVcLSxKpOthpMQES5+340o3cBBYawBRX13x5Aurldp2dD9K2FBHfnhlJTxOI+7KT8EQvV3OXzoyyoUeTlTf7qBuiNNKt+LkmESgyiNJX7nDYbvaphyY3k7L4lijRNLekfioRXzkFaAadTnezDiQ1qEtGJBIlsMHro02y2LueZiOF9i33HBvjX0dXp2J9ALIHEhCG/ZyII3IrgNHRzjz4CIcHLVQIe2mteLUH2MoNdTiq3Cg15PuQ4ZJLQnspMjrHCLYKBxJe9OD/jqe9IH66q6Uo8Zw5PJzFVWNZc1yVjsWBFH+BoP6u42Cas= ceph-4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:16.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcEMzNqK2HG7a70Uh6k8fWeYPIXoQsdPC6r3TMNdB6mcuHk2pKfUGbvCgFZa5/UA6PFfkUWV04eyZEyeqa3MrTDdSIjjMdrjp4CBgHV0PYCQYbAD0p0qUpijkgtKn9cRnQ70W6bNdHJYNoA907C9PLZ0/J9vOdiCDsQpGOiCRmx6Nmp3YV2fiugkFPexHBJ7i9WpUwfiMVcLSxKpOthpMQES5+340o3cBBYawBRX13x5Aurldp2dD9K2FBHfnhlJTxOI+7KT8EQvV3OXzoyyoUeTlTf7qBuiNNKt+LkmESgyiNJX7nDYbvaphyY3k7L4lijRNLekfioRXzkFaAadTnezDiQ1qEtGJBIlsMHro02y2LueZiOF9i33HBvjX0dXp2J9ALIHEhCG/ZyII3IrgNHRzjz4CIcHLVQIe2mteLUH2MoNdTiq3Cg15PuQ4ZJLQnspMjrHCLYKBxJe9OD/jqe9IH66q6Uo8Zw5PJzFVWNZc1yVjsWBFH+BoP6u42Cas= ceph-4b598dd2-7788-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:43:16.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:16 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3006603213' entity='client.admin' 2024-09-20T19:43:16.485 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcEMzNqK2HG7a70Uh6k8fWeYPIXoQsdPC6r3TMNdB6mcuHk2pKfUGbvCgFZa5/UA6PFfkUWV04eyZEyeqa3MrTDdSIjjMdrjp4CBgHV0PYCQYbAD0p0qUpijkgtKn9cRnQ70W6bNdHJYNoA907C9PLZ0/J9vOdiCDsQpGOiCRmx6Nmp3YV2fiugkFPexHBJ7i9WpUwfiMVcLSxKpOthpMQES5+340o3cBBYawBRX13x5Aurldp2dD9K2FBHfnhlJTxOI+7KT8EQvV3OXzoyyoUeTlTf7qBuiNNKt+LkmESgyiNJX7nDYbvaphyY3k7L4lijRNLekfioRXzkFaAadTnezDiQ1qEtGJBIlsMHro02y2LueZiOF9i33HBvjX0dXp2J9ALIHEhCG/ZyII3IrgNHRzjz4CIcHLVQIe2mteLUH2MoNdTiq3Cg15PuQ4ZJLQnspMjrHCLYKBxJe9OD/jqe9IH66q6Uo8Zw5PJzFVWNZc1yVjsWBFH+BoP6u42Cas= ceph-4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:16.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T19:43:16.819 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:19.146 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:19 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:20.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:20 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2930270960' entity='client.admin' 2024-09-20T19:43:20.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:20 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:20.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:20 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:20.551 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T19:43:20.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T19:43:21.081 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:21.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:21 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:21.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:21 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:23.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:24.010 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-09-20T19:43:24.010 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:43:24.010 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-09-20T19:43:24.030 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:43:24.030 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:24.088 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-09-20T19:43:24.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch host add smithi196 2024-09-20T19:43:24.288 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:25.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:25.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:25 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:26 smithi184 ceph-mon[23692]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:27 smithi184 ceph-mon[23692]: Deploying cephadm binary to smithi196 2024-09-20T19:43:28.468 INFO:teuthology.orchestra.run.smithi184.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-09-20T19:43:28.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:28 smithi184 ceph-mon[23692]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:29.062 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T19:43:29.258 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: Added host smithi196 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:43:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:29 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:31.005 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:43:31.006 INFO:teuthology.orchestra.run.smithi184.stdout:[{"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]': finished 2024-09-20T19:43:31.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:30 smithi184 ceph-mon[23692]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:31.603 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T19:43:31.603 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T19:43:31.798 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:31.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:43:31.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:31.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:31.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:32 smithi184 ceph-mon[23692]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:34.050 INFO:teuthology.orchestra.run.smithi184.stderr:adjusted tunables profile to default 2024-09-20T19:43:34.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:34.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T19:43:34.678 INFO:tasks.cephadm:Adding mon.a on smithi184 2024-09-20T19:43:34.678 INFO:tasks.cephadm:Adding mon.c on smithi184 2024-09-20T19:43:34.678 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-09-20T19:43:34.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b' 2024-09-20T19:43:34.886 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:35 smithi184 ceph-mon[23692]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:35 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:43:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:35 smithi184 ceph-mon[23692]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:35 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:35 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:36.493 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-09-20T19:43:37.084 DEBUG:teuthology.orchestra.run.smithi184:mon.c> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T19:43:37.087 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T19:43:37.089 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:43:37.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:37.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:37.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:37.819 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:38.155 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:38.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-20T19:43:38.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: Deploying daemon mon.b on smithi196 2024-09-20T19:43:38.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:38.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:38.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:39 smithi184 ceph-mon[23692]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:39.490 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:43:39.490 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","modified":"2024-09-20T19:42:15.806713Z","created":"2024-09-20T19:42:15.806713Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:43:39.490 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-09-20T19:43:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:40 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/1796143150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:41 smithi184 ceph-mon[23692]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:41 smithi196 systemd[1]: Starting Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:41 smithi196 podman[25643]: 2024-09-20 19:43:41.885757483 +0000 UTC m=+0.015877632 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:41 smithi196 podman[25643]: 2024-09-20 19:43:41.998261006 +0000 UTC m=+0.128381136 container create 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 podman[25643]: 2024-09-20 19:43:42.208942052 +0000 UTC m=+0.339062205 container init 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 podman[25643]: 2024-09-20 19:43:42.212068429 +0000 UTC m=+0.342188563 container start 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T19:43:42.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pidfile_write: ignore empty --pid-file 2024-09-20T19:43:42.383 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:43:42.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:43:42.595 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: load: jerasure load: lrc 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Git sha 0 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: DB SUMMARY 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: DB Session ID: 2OGZ711B262F7H3RPUXP 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:43:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.env: 0x561bb2688bc0 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.info_log: 0x561bb437abe0 2024-09-20T19:43:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.statistics: (nil) 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.use_fsync: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:43:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.db_log_dir: 2024-09-20T19:43:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.wal_dir: 2024-09-20T19:43:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:43:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:43:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:43:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:43:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:43:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:43:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:43:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:43:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.write_buffer_manager: 0x561bb4388320 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.unordered_write: 0 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:43:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.row_cache: None 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.wal_filter: None 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.wal_compression: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:43:42.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:43:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_open_files: -1 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Compression algorithms supported: 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kZSTD supported: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:43:42.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.merge_operator: 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_filter: None 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561bb437a7e0) 2024-09-20T19:43:42.649 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x561bb436f350 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-20T19:43:42.650 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-20T19:43:42.651 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression: NoCompression 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.num_levels: 7 2024-09-20T19:43:42.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:43:42.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:43:42.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:43:42.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:43:42.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.ttl: 2592000 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:43:42.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46b47607-e812-4bb9-9259-6aefaf96e6d2 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861422238754, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861422239168, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726861422, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46b47607-e812-4bb9-9259-6aefaf96e6d2", "db_session_id": "2OGZ711B262F7H3RPUXP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861422239262, "job": 1, "event": "recovery_finished"} 2024-09-20T19:43:42.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561bb439ae00 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: DB pointer 0x561bb448e000 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:43:42.659 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:43:42.660 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x561bb436f350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-20T19:43:42.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: starting mon.b rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(???) e0 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).mds e1 new map 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T19:43:42.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]': finished 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:42.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: Deploying daemon mon.b on smithi196 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/1796143150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 ceph-mon[25662]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 bash[25643]: 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 2024-09-20T19:43:42.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:42 smithi196 systemd[1]: Started Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:43:47.522 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:43:47.522 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","modified":"2024-09-20T19:43:42.264279Z","created":"2024-09-20T19:42:15.806713Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T19:43:47.522 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: mon.a calling monitor election 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: mon.b calling monitor election 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: fsmap 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: mgrmap e13: y(active, since 39s) 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: overall HEALTH_OK 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:47.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:47 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:47.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:47.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: mon.a calling monitor election 2024-09-20T19:43:47.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: mon.b calling monitor election 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: fsmap 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: mgrmap e13: y(active, since 39s) 2024-09-20T19:43:47.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: overall HEALTH_OK 2024-09-20T19:43:47.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:47.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:47.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:47 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:48 smithi196 ceph-mon[25662]: Deploying daemon mon.c on smithi184 2024-09-20T19:43:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:48 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/3753492120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:48 smithi196 ceph-mon[25662]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:48.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:48 smithi184 ceph-mon[23692]: Deploying daemon mon.c on smithi184 2024-09-20T19:43:48.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:48 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/3753492120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:48.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:48 smithi184 ceph-mon[23692]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:48.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:49.052 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T19:43:49.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:43:49.256 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:43:49.544 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:43:49 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:43:49.264+0000 7f8f60d4d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-20T19:43:50.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:49 smithi184 systemd[1]: Starting Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:43:50.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 podman[30000]: 2024-09-20 19:43:50.179739239 +0000 UTC m=+0.105003106 container create 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-09-20T19:43:50.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 podman[30000]: 2024-09-20 19:43:50.090418739 +0000 UTC m=+0.015682570 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 podman[30000]: 2024-09-20 19:43:50.382056567 +0000 UTC m=+0.307320399 container init 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d) 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 podman[30000]: 2024-09-20 19:43:50.38507635 +0000 UTC m=+0.310340175 container start 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pidfile_write: ignore empty --pid-file 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: load: jerasure load: lrc 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Git sha 0 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: DB SUMMARY 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: DB Session ID: 0VYAINIP073V2LP2ME03 2024-09-20T19:43:50.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:43:50.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:43:50.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T19:43:50.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-20T19:43:50.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-20T19:43:50.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.env: 0x55b373bedbc0 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.info_log: 0x55b375178be0 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.statistics: (nil) 2024-09-20T19:43:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.use_fsync: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.db_log_dir: 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.wal_dir: 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:43:50.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.write_buffer_manager: 0x55b375186320 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:43:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.unordered_write: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.row_cache: None 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.wal_filter: None 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.wal_compression: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:43:50.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:43:50.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_open_files: -1 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Compression algorithms supported: 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kZSTD supported: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:43:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.merge_operator: 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_filter: None 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:43:50.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b3751787e0) 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_type: 0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_index_type: 0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_shortening: 1 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: checksum: 4 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: no_block_cache: 0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache: 0x55b37516d1f0 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_options: 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: capacity : 536870912 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_shard_bits : 4 2024-09-20T19:43:50.442 INFO:journalctl@ceph.mon.c.smithi184.stdout: strict_capacity_limit : 0 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_compressed: (nil) 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: persistent_cache: (nil) 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size: 4096 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size_deviation: 10 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_restart_interval: 16 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_block_restart_interval: 1 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: metadata_block_size: 4096 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: partition_filters: 0 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: use_delta_encoding: 1 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: filter_policy: bloomfilter 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: whole_key_filtering: 1 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: verify_compression: 0 2024-09-20T19:43:50.443 INFO:journalctl@ceph.mon.c.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: format_version: 5 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_index_compression: 1 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_align: 0 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: prepopulate_block_cache: 0 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression: NoCompression 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.num_levels: 7 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:43:50.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:43:50.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:43:50.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:43:50.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:43:50.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.ttl: 2592000 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:43:50.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 88c00575-24f4-43bb-9171-52330b599b14 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861430411440, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861430411820, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726861430, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "88c00575-24f4-43bb-9171-52330b599b14", "db_session_id": "0VYAINIP073V2LP2ME03", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726861430411912, "job": 1, "event": "recovery_finished"} 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b375198e00 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: DB pointer 0x55b37528c000 2024-09-20T19:43:50.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** DB Stats ** 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:43:50.451 INFO:journalctl@ceph.mon.c.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:43:50.452 INFO:journalctl@ceph.mon.c.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache BinnedLRUCache@0x55b37516d1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: using public_addrv [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] 2024-09-20T19:43:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: starting mon.c rank -1 at public addrs [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] at bind addrs [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(???) e0 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).mds e1 new map 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: e1 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: legacy client fscid: -1 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:43:50.454 INFO:journalctl@ceph.mon.c.smithi184.stdout: No filesystems configured 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:43:50.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]': finished 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:50.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3947559665' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi184:[v2:172.21.15.184:3301,v1:172.21.15.184:6790]=c;smithi196:172.21.15.196=b;count:3 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Deploying daemon mon.b on smithi196 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/1796143150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.a calling monitor election 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.b calling monitor election 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T19:43:50.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: fsmap 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mgrmap e13: y(active, since 39s) 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: overall HEALTH_OK 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: Deploying daemon mon.c on smithi184 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/3753492120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:50.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 ceph-mon[30019]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T19:43:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 bash[30000]: 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 2024-09-20T19:43:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:50 smithi184 systemd[1]: Started Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:43:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: mon.a calling monitor election 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: mon.b calling monitor election 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: mon.c calling monitor election 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: fsmap 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: mgrmap e13: y(active, since 47s) 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: overall HEALTH_OK 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: mon.a calling monitor election 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: mon.b calling monitor election 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: mon.c calling monitor election 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: fsmap 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: mgrmap e13: y(active, since 47s) 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: overall HEALTH_OK 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: mon.a calling monitor election 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: mon.b calling monitor election 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: mon.c calling monitor election 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: fsmap 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: mgrmap e13: y(active, since 47s) 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: overall HEALTH_OK 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:56.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:56.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:43:56.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:56.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:43:56.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T19:43:56.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:43:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:43:57.015 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:43:57.016 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","modified":"2024-09-20T19:43:50.434988Z","created":"2024-09-20T19:42:15.806713Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3301","nonce":0},{"type":"v1","addr":"172.21.15.184:6790","nonce":0}]},"addr":"172.21.15.184:6790/0","public_addr":"172.21.15.184:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-20T19:43:57.016 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-09-20T19:43:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:57 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/672448012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:57 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:57.675 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T19:43:57.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T19:43:57.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:57 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/672448012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:57.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:57 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:57.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:57 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/672448012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:43:57.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:57 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:58.286 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:59.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:58 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:43:59.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:58 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:00.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[23692]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:44:00.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[23692]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:44:00.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[30019]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:44:00.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[30019]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:44:00.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:43:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:59 smithi196 ceph-mon[25662]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:44:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:59 smithi196 ceph-mon[25662]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:44:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:43:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:00 smithi184 ceph-mon[23692]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:00 smithi184 ceph-mon[30019]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:00 smithi196 ceph-mon[25662]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:02.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:44:02.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[23692]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:44:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:44:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:44:02.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:02.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:44:02.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:02 smithi184 ceph-mon[30019]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:02.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:02.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:44:02.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:44:02.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:44:02.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:02.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:44:02.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:02 smithi196 ceph-mon[25662]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:03.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:03.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:03.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:03.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:03 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:03.966 INFO:teuthology.orchestra.run.smithi184.stdout:# minimal ceph.conf for 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:44:03.966 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2024-09-20T19:44:03.966 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:44:03.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon_host = [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:03 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:04.748 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T19:44:04.748 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:44:04.748 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:44:04.822 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:44:04.823 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:44:04.890 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:44:04.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:44:04.923 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:44:04.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:44:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3703593040' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:04.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:04 smithi196 ceph-mon[25662]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:04.993 INFO:tasks.cephadm:Adding mgr.y on smithi184 2024-09-20T19:44:04.993 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-09-20T19:44:04.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi184=y;smithi196=x' 2024-09-20T19:44:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3703593040' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:04 smithi184 ceph-mon[23692]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3703593040' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:04 smithi184 ceph-mon[30019]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:05.227 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:44:06.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:06 smithi196 ceph-mon[25662]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[23692]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:06 smithi184 ceph-mon[30019]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:06.795 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-09-20T19:44:07.411 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T19:44:07.413 DEBUG:tasks.cephadm:set 0 configs 2024-09-20T19:44:07.414 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T19:44:07.414 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:44:07.414 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:44:07.433 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:44:07.433 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-09-20T19:44:07.493 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:44:07.493 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:07.493 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-20 19:44:03.834865951 +0000 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-20 19:40:49.874710310 +0000 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-20 19:40:49.874710310 +0000 2024-09-20T19:44:07.494 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-20 19:40:49.874710310 +0000 2024-09-20T19:44:07.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:44:07.564 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-20T19:44:07.564 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-20T19:44:07.564 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000237203 s, 2.2 MB/s 2024-09-20T19:44:07.565 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:44:07.624 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-09-20T19:44:07.683 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:44:07.683 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:07.683 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-20T19:44:07.683 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:07.683 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:07.684 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-20 19:44:03.834865951 +0000 2024-09-20T19:44:07.684 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-20 19:40:50.206703545 +0000 2024-09-20T19:44:07.684 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-20 19:40:50.206703545 +0000 2024-09-20T19:44:07.684 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-20 19:40:50.206703545 +0000 2024-09-20T19:44:07.684 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:44:07.751 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-20T19:44:07.751 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-20T19:44:07.751 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00030638 s, 1.7 MB/s 2024-09-20T19:44:07.753 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:44:07.811 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-09-20T19:44:07.869 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:44:07.869 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:07.869 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-20T19:44:07.869 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:07.870 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:07.870 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-20 19:44:03.834865951 +0000 2024-09-20T19:44:07.870 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-20 19:40:50.533696882 +0000 2024-09-20T19:44:07.870 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-20 19:40:50.533696882 +0000 2024-09-20T19:44:07.870 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-20 19:40:50.533696882 +0000 2024-09-20T19:44:07.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi184=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: Saving service mgr spec with placement smithi184=y;smithi196=x;count:2 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[23692]: Deploying daemon mgr.x on smithi196 2024-09-20T19:44:07.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi184=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: Saving service mgr spec with placement smithi184=y;smithi196=x;count:2 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:07.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:07 smithi184 ceph-mon[30019]: Deploying daemon mgr.x on smithi196 2024-09-20T19:44:07.935 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-20T19:44:07.935 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-20T19:44:07.935 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00025375 s, 2.0 MB/s 2024-09-20T19:44:07.937 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:44:07.995 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-09-20T19:44:08.051 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-20 19:44:03.834865951 +0000 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-20 19:40:50.867690077 +0000 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-20 19:40:50.867690077 +0000 2024-09-20T19:44:08.052 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-09-20 19:40:50.867690077 +0000 2024-09-20T19:44:08.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:44:08.118 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-20T19:44:08.118 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-20T19:44:08.118 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000245539 s, 2.1 MB/s 2024-09-20T19:44:08.120 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:44:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi184=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: Saving service mgr spec with placement smithi184=y;smithi196=x;count:2 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:07 smithi196 ceph-mon[25662]: Deploying daemon mgr.x on smithi196 2024-09-20T19:44:08.177 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:44:08.178 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:44:08.194 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:44:08.194 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-09-20T19:44:08.249 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:44:08.249 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:08.249 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-20T19:44:08.249 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-20 19:43:58.766341240 +0000 2024-09-20T19:44:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-20 19:40:54.662574740 +0000 2024-09-20T19:44:08.250 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-20 19:40:54.662574740 +0000 2024-09-20T19:44:08.250 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-20 19:40:54.662574740 +0000 2024-09-20T19:44:08.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:44:08.313 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-20T19:44:08.313 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-20T19:44:08.313 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000282358 s, 1.8 MB/s 2024-09-20T19:44:08.315 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:44:08.371 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-09-20T19:44:08.426 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:44:08.426 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:08.426 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-20T19:44:08.426 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:08.426 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:08.427 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-20 19:43:58.766341240 +0000 2024-09-20T19:44:08.427 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-20 19:40:55.002567651 +0000 2024-09-20T19:44:08.427 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-20 19:40:55.002567651 +0000 2024-09-20T19:44:08.427 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-20 19:40:55.002567651 +0000 2024-09-20T19:44:08.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:44:08.490 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-20T19:44:08.490 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-20T19:44:08.490 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000177029 s, 2.9 MB/s 2024-09-20T19:44:08.491 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:44:08.546 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-09-20T19:44:08.612 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:44:08.612 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:08.612 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-20T19:44:08.612 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:08.612 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:08.613 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-20 19:43:58.766341240 +0000 2024-09-20T19:44:08.613 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-20 19:40:55.323560958 +0000 2024-09-20T19:44:08.613 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-20 19:40:55.323560958 +0000 2024-09-20T19:44:08.613 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-20 19:40:55.323560958 +0000 2024-09-20T19:44:08.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:44:08.724 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-20T19:44:08.724 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-20T19:44:08.724 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000214191 s, 2.4 MB/s 2024-09-20T19:44:08.725 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:44:08.740 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-20 19:44:08.674134377 +0000 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-20 19:40:55.695553203 +0000 2024-09-20T19:44:08.802 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-20 19:40:55.695553203 +0000 2024-09-20T19:44:08.803 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-20 19:40:55.695553203 +0000 2024-09-20T19:44:08.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:44:08.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:08 smithi196 ceph-mon[25662]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:08.908 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-20T19:44:08.909 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-20T19:44:08.909 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000200959 s, 2.5 MB/s 2024-09-20T19:44:08.910 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:44:08.925 INFO:tasks.cephadm:Deploying osd.0 on smithi184 with /dev/vg_nvme/lv_4... 2024-09-20T19:44:08.926 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:44:09.134 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:09.140 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:08 smithi196 systemd[1]: Starting Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:44:09.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:08 smithi184 ceph-mon[23692]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:09.199 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:08 smithi184 ceph-mon[30019]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:09.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 podman[27867]: 2024-09-20 19:44:09.045236029 +0000 UTC m=+0.015659170 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T19:44:09.396 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 podman[27867]: 2024-09-20 19:44:09.157621641 +0000 UTC m=+0.128044816 container create 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T19:44:09.396 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 podman[27867]: 2024-09-20 19:44:09.367195702 +0000 UTC m=+0.337618838 container init 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:44:09.396 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 podman[27867]: 2024-09-20 19:44:09.369974186 +0000 UTC m=+0.340397316 container start 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0) 2024-09-20T19:44:09.396 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-mgr[27887]: -- 172.21.15.196:0/296565252 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5562bc2d2d00 con 0x5562bc2a6800 2024-09-20T19:44:09.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 bash[27867]: 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f 2024-09-20T19:44:09.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 systemd[1]: Started Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:44:09.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:09.492+0000 7fb0a75dd040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:44:09.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:09.552+0000 7fb0a75dd040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:44:09.711 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:09.708+0000 7fb0a75dd040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:44:10.056 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:09.774+0000 7fb0a75dd040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:44:10.056 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:09.918+0000 7fb0a75dd040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:44:10.056 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:10.054+0000 7fb0a75dd040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:44:10.634 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:10.380+0000 7fb0a75dd040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:44:10.634 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:10.441+0000 7fb0a75dd040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:44:10.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:44:10.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-mon[25662]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[23692]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:44:10.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:10 smithi184 ceph-mon[30019]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:10.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:10.632+0000 7fb0a75dd040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:44:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:44:11.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-09-20T19:44:11.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:11.536+0000 7fb0a75dd040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:44:11.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:11.605+0000 7fb0a75dd040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:44:12.639 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.414+0000 7fb0a75dd040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:44:12.639 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.530+0000 7fb0a75dd040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:44:12.660 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:12.989 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.680+0000 7fb0a75dd040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:44:12.990 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.749+0000 7fb0a75dd040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:44:12.990 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.918+0000 7fb0a75dd040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:44:12.990 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:12.988+0000 7fb0a75dd040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.047+0000 7fb0a75dd040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.211+0000 7fb0a75dd040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:13.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:13.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:13.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:13.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:13.547 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.282+0000 7fb0a75dd040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:44:13.548 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.346+0000 7fb0a75dd040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:44:13.548 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.429+0000 7fb0a75dd040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:44:13.548 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.488+0000 7fb0a75dd040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:44:13.548 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:13.546+0000 7fb0a75dd040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.640 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:44:14 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:44:14.325+0000 7fb0a75dd040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:44:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:14 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:44:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: Standby manager daemon x started 2024-09-20T19:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:44:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:15 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:15 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/2219866531' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:44:16.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:16 smithi196 ceph-mon[25662]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:16.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:16 smithi196 ceph-mon[25662]: mgrmap e14: y(active, since 67s), standbys: x 2024-09-20T19:44:16.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:16 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:44:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[23692]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[23692]: mgrmap e14: y(active, since 67s), standbys: x 2024-09-20T19:44:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:44:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[30019]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[30019]: mgrmap e14: y(active, since 67s), standbys: x 2024-09-20T19:44:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:16 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1531539265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]': finished 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/370827829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1531539265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]': finished 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/370827829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1531539265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]: dispatch 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18242be3-3630-4599-8612-ba05cd943947"}]': finished 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/370827829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:18 smithi184 ceph-mon[23692]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:18 smithi184 ceph-mon[30019]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:18 smithi196 ceph-mon[25662]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:21 smithi184 ceph-mon[23692]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:21 smithi184 ceph-mon[30019]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:21 smithi196 ceph-mon[25662]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:23.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[23692]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:23.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T19:44:23.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:23.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[30019]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:23.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T19:44:23.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:23 smithi196 ceph-mon[25662]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T19:44:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:24.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:24 smithi184 ceph-mon[23692]: Deploying daemon osd.0 on smithi184 2024-09-20T19:44:24.256 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:24 smithi184 ceph-mon[30019]: Deploying daemon osd.0 on smithi184 2024-09-20T19:44:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:24 smithi196 ceph-mon[25662]: Deploying daemon osd.0 on smithi184 2024-09-20T19:44:25.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:25 smithi184 ceph-mon[23692]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:25.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:25 smithi184 ceph-mon[30019]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:25 smithi196 ceph-mon[25662]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:26 smithi196 ceph-mon[25662]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[23692]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:27.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:26 smithi184 ceph-mon[30019]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:28 smithi184 ceph-mon[23692]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:28 smithi184 ceph-mon[30019]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:28 smithi196 ceph-mon[25662]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:31.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:31 smithi196 ceph-mon[25662]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:31.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:31 smithi196 ceph-mon[25662]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:31 smithi196 ceph-mon[25662]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[23692]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:31.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[23692]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[23692]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[30019]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:31.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[30019]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:31 smithi184 ceph-mon[30019]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:44:31.438 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 0 on host 'smithi184' 2024-09-20T19:44:31.994 DEBUG:teuthology.orchestra.run.smithi184:osd.0> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T19:44:31.996 INFO:tasks.cephadm:Deploying osd.1 on smithi184 with /dev/vg_nvme/lv_3... 2024-09-20T19:44:31.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:44:32.233 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:44:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:32.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:32 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:32.668 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[34428]: 2024-09-20T19:44:32.526+0000 7f55078e3640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T19:44:32.668 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[34428]: 2024-09-20T19:44:32.530+0000 7f54ff704640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:44:33.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi184 to 15040M 2024-09-20T19:44:33.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:33.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:33.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:44:33.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:33 smithi196 ceph-mon[25662]: from='osd.0 ' entity='osd.0' 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi184 to 15040M 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[30019]: from='osd.0 ' entity='osd.0' 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi184 to 15040M 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:33 smithi184 ceph-mon[23692]: from='osd.0 ' entity='osd.0' 2024-09-20T19:44:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:44:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:44:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610] boot 2024-09-20T19:44:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:44:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610] boot 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:44:34.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:34 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: osd.0 [v2:172.21.15.184:6802/3004532610,v1:172.21.15.184:6803/3004532610] boot 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T19:44:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:34 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:44:34.549 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:44:34.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-09-20T19:44:34.762 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:35.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:35 smithi184 ceph-mon[23692]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:35.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:35 smithi184 ceph-mon[23692]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:44:35.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:35 smithi184 ceph-mon[30019]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:35.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:35 smithi184 ceph-mon[30019]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:44:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:35 smithi196 ceph-mon[25662]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:44:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:35 smithi196 ceph-mon[25662]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[23692]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[30019]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:37 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:37 smithi196 ceph-mon[25662]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:37 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:37 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:37 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:38 smithi196 ceph-mon[25662]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:38 smithi184 ceph-mon[23692]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:38 smithi184 ceph-mon[30019]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:39 smithi196 ceph-mon[25662]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:39 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]: dispatch 2024-09-20T19:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:39 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]': finished 2024-09-20T19:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:39 smithi196 ceph-mon[25662]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:39 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[23692]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]: dispatch 2024-09-20T19:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]': finished 2024-09-20T19:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[23692]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[30019]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]: dispatch 2024-09-20T19:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2398423437' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5d9b3e3-baf4-4c10-bec1-99c167584ae3"}]': finished 2024-09-20T19:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[30019]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T19:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:39 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:40.318 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:40 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3180543623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:40.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:40 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3180543623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:40 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3180543623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:44:41.282 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:41 smithi184 ceph-mon[23692]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:41.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:41 smithi184 ceph-mon[30019]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:41 smithi196 ceph-mon[25662]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[23692]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:44:42.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:42 smithi184 ceph-mon[30019]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:42 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:44:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:42 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:42 smithi196 ceph-mon[25662]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:45 smithi196 ceph-mon[25662]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T19:44:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:45.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[23692]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:45.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T19:44:45.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:45.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[30019]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:45.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T19:44:45.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:46 smithi196 ceph-mon[25662]: Deploying daemon osd.1 on smithi184 2024-09-20T19:44:46.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:46 smithi184 ceph-mon[23692]: Deploying daemon osd.1 on smithi184 2024-09-20T19:44:46.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:46 smithi184 ceph-mon[30019]: Deploying daemon osd.1 on smithi184 2024-09-20T19:44:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:47 smithi184 ceph-mon[23692]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:47 smithi184 ceph-mon[30019]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:47 smithi196 ceph-mon[25662]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:49 smithi184 ceph-mon[23692]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:49 smithi184 ceph-mon[30019]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:49 smithi196 ceph-mon[25662]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:51.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:51 smithi184 ceph-mon[23692]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:51.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:51 smithi184 ceph-mon[30019]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:51 smithi196 ceph-mon[25662]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:53.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[23692]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:53.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[30019]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:53.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:53 smithi196 ceph-mon[25662]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:53.513 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 1 on host 'smithi184' 2024-09-20T19:44:54.220 DEBUG:teuthology.orchestra.run.smithi184:osd.1> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T19:44:54.223 INFO:tasks.cephadm:Deploying osd.2 on smithi184 with /dev/vg_nvme/lv_2... 2024-09-20T19:44:54.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:54.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:44:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:44:54.462 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi184 to 7520M 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[30019]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi184 to 7520M 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-mon[23692]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi184 to 7520M 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: from='osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:55 smithi196 ceph-mon[25662]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:44:55.540 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[37787]: 2024-09-20T19:44:55.478+0000 7f5897051640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T19:44:55.540 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:44:55 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[37787]: 2024-09-20T19:44:55.483+0000 7f588ee72640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[23692]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[23692]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[23692]: from='osd.1 ' entity='osd.1' 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[30019]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:56.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[30019]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:44:56.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:56 smithi184 ceph-mon[30019]: from='osd.1 ' entity='osd.1' 2024-09-20T19:44:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:56 smithi196 ceph-mon[25662]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:44:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:56 smithi196 ceph-mon[25662]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:44:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:56 smithi196 ceph-mon[25662]: from='osd.1 ' entity='osd.1' 2024-09-20T19:44:56.883 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:44:56.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-09-20T19:44:57.144 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109] boot 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:57.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109] boot 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:44:57.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:57 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: osd.1 [v2:172.21.15.184:6810/3361797109,v1:172.21.15.184:6811/3361797109] boot 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T19:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:57 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:44:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:59 smithi196 ceph-mon[25662]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:59 smithi196 ceph-mon[25662]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:44:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:59.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[30019]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:44:59.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[30019]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:44:59.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[23692]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[23692]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:44:59.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:44:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:00 smithi196 ceph-mon[25662]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:00.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:00 smithi184 ceph-mon[23692]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:00.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:00 smithi184 ceph-mon[30019]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:01 smithi184 ceph-mon[23692]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:01 smithi184 ceph-mon[30019]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:01 smithi196 ceph-mon[25662]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:02.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]: dispatch 2024-09-20T19:45:02.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]': finished 2024-09-20T19:45:02.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1907553426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]: dispatch 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]': finished 2024-09-20T19:45:02.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1907553426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/415359624' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0775d00f-951a-431d-9686-6531ffa73c4d"}]': finished 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1907553426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:45:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:03.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[23692]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:45:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[30019]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:04 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:45:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:04 smithi196 ceph-mon[25662]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:04 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:04 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:45:06.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:06 smithi184 ceph-mon[23692]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:06.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:06 smithi184 ceph-mon[30019]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:06 smithi196 ceph-mon[25662]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:08.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:08 smithi184 ceph-mon[23692]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:08.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:08 smithi184 ceph-mon[30019]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:08 smithi196 ceph-mon[25662]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T19:45:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[23692]: Deploying daemon osd.2 on smithi184 2024-09-20T19:45:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T19:45:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:09 smithi184 ceph-mon[30019]: Deploying daemon osd.2 on smithi184 2024-09-20T19:45:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T19:45:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:09 smithi196 ceph-mon[25662]: Deploying daemon osd.2 on smithi184 2024-09-20T19:45:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:10 smithi184 ceph-mon[23692]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:10 smithi184 ceph-mon[30019]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:10 smithi196 ceph-mon[25662]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:12.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:12 smithi184 ceph-mon[23692]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:12.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:12 smithi184 ceph-mon[30019]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:12.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:12 smithi196 ceph-mon[25662]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:14 smithi196 ceph-mon[25662]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:14.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:14 smithi184 ceph-mon[23692]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:14.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:14 smithi184 ceph-mon[30019]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:15 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:15 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:16.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:16 smithi184 ceph-mon[23692]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:16.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:16 smithi184 ceph-mon[30019]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:16.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:16 smithi196 ceph-mon[25662]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:17.124 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 2 on host 'smithi184' 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:17.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:17.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:17.753 DEBUG:teuthology.orchestra.run.smithi184:osd.2> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T19:45:17.755 INFO:tasks.cephadm:Deploying osd.3 on smithi184 with /dev/vg_nvme/lv_1... 2024-09-20T19:45:17.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T19:45:17.986 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[23692]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi184 to 5013M 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[23692]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[23692]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[30019]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi184 to 5013M 2024-09-20T19:45:18.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[30019]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:18.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:18 smithi184 ceph-mon[30019]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:18 smithi196 ceph-mon[25662]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:18 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi184 to 5013M 2024-09-20T19:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:18 smithi196 ceph-mon[25662]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:18 smithi196 ceph-mon[25662]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[23692]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[23692]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[23692]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[23692]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[30019]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[30019]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[30019]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-mon[30019]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:19 smithi196 ceph-mon[25662]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:19 smithi196 ceph-mon[25662]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:19 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:19 smithi196 ceph-mon[25662]: from='osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:19 smithi196 ceph-mon[25662]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:20.049 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[41348]: 2024-09-20T19:45:19.660+0000 7fd259d4e640 -1 osd.2 0 waiting for initial osdmap 2024-09-20T19:45:20.049 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[41348]: 2024-09-20T19:45:19.663+0000 7fd251b6f640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:45:20.254 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:45:20.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi184:vg_nvme/lv_1 2024-09-20T19:45:20.468 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[23692]: from='osd.2 ' entity='osd.2' 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:20 smithi184 ceph-mon[30019]: from='osd.2 ' entity='osd.2' 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:20 smithi196 ceph-mon[25662]: from='osd.2 ' entity='osd.2' 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898] boot 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T19:45:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898] boot 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T19:45:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:21 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: osd.2 [v2:172.21.15.184:6818/795619898,v1:172.21.15.184:6819/795619898] boot 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T19:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:21 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:45:22.571 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:22 smithi184 ceph-mon[23692]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:22.571 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:22 smithi184 ceph-mon[23692]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:45:22.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:22 smithi184 ceph-mon[30019]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:22.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:22 smithi184 ceph-mon[30019]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:45:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:22 smithi196 ceph-mon[25662]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:45:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:22 smithi196 ceph-mon[25662]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:24.088 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42949]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:24.088 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42949]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:24.088 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42949]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:24.342 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42949]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:24.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[23692]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:24.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[23692]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[23692]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[30019]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[30019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:24.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:24 smithi184 ceph-mon[30019]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:45:24.610 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42956]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:24.610 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42956]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:24.611 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42956]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:24.611 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42956]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:24 smithi196 ceph-mon[25662]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:45:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:24 smithi196 ceph-mon[25662]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:45:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:45:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:24 smithi196 ceph-mon[25662]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T19:45:24.875 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42959]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:24.875 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42959]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:24.875 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42959]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:24.875 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42959]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:25.125 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42981]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:25.126 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42981]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:25.126 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42981]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:25.126 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:45:24 smithi184 sudo[42981]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:25.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:25.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[23692]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T19:45:25.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[23692]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:45:25.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[23692]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:25.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:25.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[30019]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T19:45:25.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[30019]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:45:25.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:25 smithi184 ceph-mon[30019]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:25.396 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[42989]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:25.396 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[42989]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:25.396 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[42989]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:25.396 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[42989]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:25 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:25 smithi196 ceph-mon[25662]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T19:45:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:25 smithi196 ceph-mon[25662]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T19:45:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:25 smithi196 ceph-mon[25662]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:25.656 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:25.656 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:25.656 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43012]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:25.656 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43012]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43060]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43060]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43060]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43060]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43154]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43154]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43154]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:25 smithi184 sudo[43154]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:26.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:26.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:26.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29221]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:26.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29221]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:26.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:26.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:26.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:45:26.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29224]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 sudo[29224]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:45:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:26.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:26.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:26.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:26.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43172]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:45:26.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43172]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:26.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43172]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:26.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43172]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43194]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:45:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43194]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:45:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43194]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:45:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:26 smithi184 sudo[43194]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3121129079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]': finished 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:27 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2811846583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:27.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.849 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3121129079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]': finished 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.850 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2811846583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:27.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3121129079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ca7dee9-4973-46bb-bd7e-93f451d719c1"}]': finished 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:27.852 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:27.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:27 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2811846583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:28 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:28 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:45:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:28 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:28 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:45:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:28 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:45:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:28 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:45:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:29 smithi184 ceph-mon[23692]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:29 smithi184 ceph-mon[30019]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:29 smithi196 ceph-mon[25662]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:31 smithi184 ceph-mon[23692]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:31 smithi184 ceph-mon[30019]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:31 smithi196 ceph-mon[25662]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[30019]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:33 smithi184 ceph-mon[23692]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:33.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:33.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:33 smithi196 ceph-mon[25662]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[23692]: Deploying daemon osd.3 on smithi184 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:34.585 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:34 smithi184 ceph-mon[30019]: Deploying daemon osd.3 on smithi184 2024-09-20T19:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:34 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T19:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:34 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:34 smithi196 ceph-mon[25662]: Deploying daemon osd.3 on smithi184 2024-09-20T19:45:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:35 smithi184 ceph-mon[23692]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:35 smithi184 ceph-mon[30019]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:35 smithi196 ceph-mon[25662]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:37 smithi184 ceph-mon[23692]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:37 smithi184 ceph-mon[30019]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:37 smithi196 ceph-mon[25662]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:38.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:38.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:38.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:39.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:39 smithi184 ceph-mon[23692]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:39.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:39 smithi184 ceph-mon[30019]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:39 smithi196 ceph-mon[25662]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:41.036 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 3 on host 'smithi184' 2024-09-20T19:45:41.573 DEBUG:teuthology.orchestra.run.smithi184:osd.3> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T19:45:41.575 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-09-20T19:45:41.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:45:41.782 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi184 to 3760M 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi184 to 3760M 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:41 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi184 to 3760M 2024-09-20T19:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:45:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:41 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:42.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:42 smithi196 ceph-mon[25662]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:42.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:42 smithi196 ceph-mon[25662]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:42 smithi184 ceph-mon[23692]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:42 smithi184 ceph-mon[23692]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:42 smithi184 ceph-mon[30019]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:42 smithi184 ceph-mon[30019]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:43 smithi196 ceph-mon[25662]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[30019]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:45:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:45:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: from='osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-mon[23692]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-09-20T19:45:43.800 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[44924]: 2024-09-20T19:45:43.539+0000 7f189c8eb640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T19:45:43.800 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:45:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[44924]: 2024-09-20T19:45:43.543+0000 7f1898714640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:45:44.094 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:45:44.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-09-20T19:45:44.309 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:45:44.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:44 smithi196 ceph-mon[25662]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:44.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:44 smithi196 ceph-mon[25662]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:45:44.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:44 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:44.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:44 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[23692]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[23692]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[30019]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-09-20T19:45:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[30019]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:45:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:44 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175] boot 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175] boot 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:45 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:45:45.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:45:45.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:45:45.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: osd.3 [v2:172.21.15.184:6826/3829952175,v1:172.21.15.184:6827/3829952175] boot 2024-09-20T19:45:45.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T19:45:45.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:45:45.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:45.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:45 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:46.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:45:46.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:45:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:47 smithi184 ceph-mon[23692]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:47 smithi184 ceph-mon[30019]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:47.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:47 smithi196 ceph-mon[25662]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:48.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:48 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/468506992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:48.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:48 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:48.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:48 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]': finished 2024-09-20T19:45:48.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:48 smithi196 ceph-mon[25662]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T19:45:48.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:45:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/468506992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]': finished 2024-09-20T19:45:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[23692]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T19:45:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:45:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/468506992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]: dispatch 2024-09-20T19:45:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8a1cfe5-33b4-4383-8ada-32ef13d3b11c"}]': finished 2024-09-20T19:45:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[30019]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T19:45:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:45:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:49 smithi196 ceph-mon[25662]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:49 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/1703048833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:49 smithi184 ceph-mon[23692]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:49 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/1703048833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:49 smithi184 ceph-mon[30019]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:49 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/1703048833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:45:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:51 smithi196 ceph-mon[25662]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:51 smithi184 ceph-mon[23692]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:52.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:51 smithi184 ceph-mon[30019]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:53.668 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:53 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.669 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:53 smithi196 ceph-mon[25662]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[23692]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:45:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:53 smithi184 ceph-mon[30019]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[23692]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[23692]: Deploying daemon osd.4 on smithi196 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[30019]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T19:45:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:55 smithi184 ceph-mon[30019]: Deploying daemon osd.4 on smithi196 2024-09-20T19:45:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:55 smithi196 ceph-mon[25662]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T19:45:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:45:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:55 smithi196 ceph-mon[25662]: Deploying daemon osd.4 on smithi196 2024-09-20T19:45:57.699 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:57 smithi196 ceph-mon[25662]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:57 smithi184 ceph-mon[23692]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:57 smithi184 ceph-mon[30019]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:59.700 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:45:59 smithi196 ceph-mon[25662]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:45:59 smithi184 ceph-mon[23692]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:45:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:45:59 smithi184 ceph-mon[30019]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:01 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:01.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:01.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:01.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:01.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:01.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:01 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.440 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:03.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:03.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:03 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:03 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:04.023 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T19:46:04.026 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-09-20T19:46:04.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:46:04.258 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:04 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi196 to 16064M 2024-09-20T19:46:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:04 smithi196 ceph-mon[25662]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:04 smithi196 ceph-mon[25662]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi196 to 16064M 2024-09-20T19:46:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[23692]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[23692]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi196 to 16064M 2024-09-20T19:46:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[30019]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:04 smithi184 ceph-mon[30019]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:05.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-mon[25662]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[23692]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: from='osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:05.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:05 smithi184 ceph-mon[30019]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:06.295 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[31898]: 2024-09-20T19:46:05.966+0000 7f8964b16640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T19:46:06.295 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 19:46:05 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[31898]: 2024-09-20T19:46:05.969+0000 7f895c937640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:46:06.628 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:46:06.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-09-20T19:46:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[30019]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[30019]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:46:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[30019]: from='osd.4 ' entity='osd.4' 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[23692]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[23692]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:06 smithi184 ceph-mon[23692]: from='osd.4 ' entity='osd.4' 2024-09-20T19:46:06.860 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:06 smithi196 ceph-mon[25662]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:06 smithi196 ceph-mon[25662]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T19:46:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:06 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:06 smithi196 ceph-mon[25662]: from='osd.4 ' entity='osd.4' 2024-09-20T19:46:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859] boot 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[23692]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859] boot 2024-09-20T19:46:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T19:46:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:07 smithi184 ceph-mon[30019]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: osd.4 [v2:172.21.15.196:6800/3871353859,v1:172.21.15.196:6801/3871353859] boot 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:46:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:07 smithi196 ceph-mon[25662]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T19:46:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T19:46:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T19:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:10 smithi196 ceph-mon[25662]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T19:46:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:10 smithi184 ceph-mon[23692]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T19:46:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:10 smithi184 ceph-mon[30019]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T19:46:11.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/4009683926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]': finished 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:11.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:11 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/1834866912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:46:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/4009683926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]': finished 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/1834866912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/4009683926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]: dispatch 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5943d045-1e02-4e0e-a0d5-0ba436281904"}]': finished 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:11 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/1834866912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi196 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi196 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi196 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 168 KiB/s, 0 objects/s recovering 2024-09-20T19:46:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:14.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:14 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:14.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:14 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:46:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:14 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:14 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:46:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:14 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:14 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:46:15.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:15 smithi196 ceph-mon[25662]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T19:46:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:15 smithi184 ceph-mon[23692]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T19:46:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:15 smithi184 ceph-mon[30019]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T19:46:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:18 smithi196 ceph-mon[25662]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-20T19:46:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:18 smithi184 ceph-mon[23692]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-20T19:46:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:18 smithi184 ceph-mon[30019]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-20T19:46:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:19 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T19:46:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:19 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:19 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T19:46:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:19 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:19.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:19 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T19:46:19.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:19 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:20 smithi196 ceph-mon[25662]: Deploying daemon osd.5 on smithi196 2024-09-20T19:46:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:20 smithi196 ceph-mon[25662]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:20 smithi184 ceph-mon[23692]: Deploying daemon osd.5 on smithi196 2024-09-20T19:46:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:20 smithi184 ceph-mon[23692]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:20 smithi184 ceph-mon[30019]: Deploying daemon osd.5 on smithi196 2024-09-20T19:46:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:20 smithi184 ceph-mon[30019]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:22 smithi184 ceph-mon[23692]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:22 smithi184 ceph-mon[30019]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:22.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:22 smithi196 ceph-mon[25662]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:24.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.046 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:26.695 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T19:46:26.698 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-09-20T19:46:26.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:46:26.929 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:27.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi196 to 8032M 2024-09-20T19:46:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:27 smithi196 ceph-mon[25662]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi196 to 8032M 2024-09-20T19:46:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[23692]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi196 to 8032M 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:27 smithi184 ceph-mon[30019]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:46:28.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:28.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:46:28.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T19:46:28.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:28.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:28.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-mon[25662]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:28.222 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[35352]: 2024-09-20T19:46:28.079+0000 7f9c12cc6640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T19:46:28.222 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 19:46:28 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[35352]: 2024-09-20T19:46:28.084+0000 7f9c0eaef640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T19:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[23692]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: from='osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:28 smithi184 ceph-mon[30019]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:29.200 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:46:29.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-09-20T19:46:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518] boot 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T19:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:29 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.421 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T19:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518] boot 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: osd.5 [v2:172.21.15.196:6808/3149907518,v1:172.21.15.196:6809/3149907518] boot 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T19:46:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:29 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:46:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:30 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:46:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:30 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:46:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:30 smithi196 ceph-mon[25662]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:30 smithi196 ceph-mon[25662]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T19:46:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:46:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:46:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[23692]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[23692]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T19:46:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:46:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:46:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[30019]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:46:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:30 smithi184 ceph-mon[30019]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T19:46:31.762 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T19:46:31.762 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:31.762 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:31.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:31.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:31.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:31 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:31 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:32.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:32 smithi196 ceph-mon[25662]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T19:46:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:32 smithi184 ceph-mon[23692]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T19:46:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:32 smithi184 ceph-mon[30019]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T19:46:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T19:46:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: pgmap v139: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi196 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: pgmap v139: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi196 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:33 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: pgmap v139: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi196 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:33 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:34.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/1896661653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]': finished 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:34.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:34 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/3241187426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/1896661653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]': finished 2024-09-20T19:46:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/3241187426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/1896661653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]: dispatch 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7844d708-111d-4284-b5f7-017b77a8cc4e"}]': finished 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:34 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/3241187426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:36.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:36.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:36.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:36 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:46:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T19:46:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:46:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:36 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:46:37.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:37 smithi196 ceph-mon[25662]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-20T19:46:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:37 smithi184 ceph-mon[30019]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-20T19:46:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:37 smithi184 ceph-mon[23692]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-20T19:46:39.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:39 smithi184 ceph-mon[23692]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T19:46:39.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:39 smithi184 ceph-mon[30019]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T19:46:39.610 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:39 smithi196 ceph-mon[25662]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T19:46:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:40 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T19:46:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:40 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:40 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T19:46:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:40 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:40 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T19:46:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:40 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:41 smithi184 ceph-mon[23692]: Deploying daemon osd.6 on smithi196 2024-09-20T19:46:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:41 smithi184 ceph-mon[23692]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:41 smithi184 ceph-mon[30019]: Deploying daemon osd.6 on smithi196 2024-09-20T19:46:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:41 smithi184 ceph-mon[30019]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:42.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:41 smithi196 ceph-mon[25662]: Deploying daemon osd.6 on smithi196 2024-09-20T19:46:42.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:41 smithi196 ceph-mon[25662]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-20T19:46:43.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:43 smithi184 ceph-mon[23692]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:46:43.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:43 smithi196 ceph-mon[25662]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:46:43.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:43 smithi184 ceph-mon[30019]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-20T19:46:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:44 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:44 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:44 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:44 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:45 smithi184 ceph-mon[23692]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:45 smithi184 ceph-mon[30019]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:45 smithi196 ceph-mon[25662]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:46 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:46 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:47 smithi196 ceph-mon[25662]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:47 smithi184 ceph-mon[23692]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:47 smithi184 ceph-mon[30019]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:48.546 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-09-20T19:46:49.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:49.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:49.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:48 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.119 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T19:46:49.122 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-09-20T19:46:49.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T19:46:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:48 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:49.356 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:49 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi196 to 5354M 2024-09-20T19:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:49 smithi196 ceph-mon[25662]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:49 smithi196 ceph-mon[25662]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:49 smithi196 ceph-mon[25662]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi196 to 5354M 2024-09-20T19:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[23692]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[23692]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[23692]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi196 to 5354M 2024-09-20T19:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[30019]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[30019]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T19:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:49 smithi184 ceph-mon[30019]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-mon[25662]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:46:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-mon[25662]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T19:46:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-mon[25662]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-mon[25662]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.052 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[38795]: 2024-09-20T19:46:50.847+0000 7fe085af0640 -1 osd.6 0 waiting for initial osdmap 2024-09-20T19:46:51.052 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 19:46:50 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[38795]: 2024-09-20T19:46:50.853+0000 7fe081919640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:46:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[23692]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:46:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[23692]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T19:46:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[23692]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[23692]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[30019]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T19:46:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[30019]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T19:46:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:51.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[30019]: from='osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:50 smithi184 ceph-mon[30019]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:46:51.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:51 smithi196 ceph-mon[25662]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:51.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:51 smithi196 ceph-mon[25662]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T19:46:51.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:51 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:51.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:51 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:51.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:51 smithi196 ceph-mon[25662]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:51.874 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-20T19:46:51.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-09-20T19:46:52.088 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:46:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[23692]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[23692]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T19:46:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[23692]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[30019]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[30019]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:51 smithi184 ceph-mon[30019]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492] boot 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T19:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:52 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492] boot 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T19:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: osd.6 [v2:172.21.15.196:6816/895991492,v1:172.21.15.196:6817/895991492] boot 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T19:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:52 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:46:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:53 smithi196 ceph-mon[25662]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T19:46:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:53 smithi196 ceph-mon[25662]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:53 smithi196 ceph-mon[25662]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T19:46:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[23692]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T19:46:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[23692]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[23692]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T19:46:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[30019]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T19:46:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[30019]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:53 smithi184 ceph-mon[30019]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T19:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:54 smithi196 ceph-mon[25662]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:54 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[23692]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[30019]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:54 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T19:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:56.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:55 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:55 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:57.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi196 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/1679863717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]': finished 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T19:46:57.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:56 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:46:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi196 2024-09-20T19:46:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T19:46:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/1679863717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]': finished 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi196 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T19:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/1679863717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]: dispatch 2024-09-20T19:46:57.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7918d00a-88b7-4a38-8442-9d2e9cc40e5b"}]': finished 2024-09-20T19:46:57.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T19:46:57.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:56 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:46:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:57 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/3725775582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:57 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/3725775582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:57 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/3725775582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:46:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:46:59 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 162 KiB/s, 0 objects/s recovering 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:46:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:46:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:46:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:46:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:46:59 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:00.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:00 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:47:00.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:00 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:47:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:00 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:47:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:00 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:47:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:00 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T19:47:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:00 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:47:01.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:01 smithi196 ceph-mon[25662]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:01 smithi184 ceph-mon[23692]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:01 smithi184 ceph-mon[30019]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:03.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:03 smithi184 ceph-mon[23692]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:03.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:03 smithi184 ceph-mon[30019]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:03.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:03 smithi196 ceph-mon[25662]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T19:47:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T19:47:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:04 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:04 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T19:47:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:04 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:04 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:04 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:04 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:05.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:05 smithi196 ceph-mon[25662]: Deploying daemon osd.7 on smithi196 2024-09-20T19:47:05.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:05 smithi196 ceph-mon[25662]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:05.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:05 smithi184 ceph-mon[23692]: Deploying daemon osd.7 on smithi196 2024-09-20T19:47:05.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:05 smithi184 ceph-mon[23692]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:05.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:05 smithi184 ceph-mon[30019]: Deploying daemon osd.7 on smithi196 2024-09-20T19:47:05.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:05 smithi184 ceph-mon[30019]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:07 smithi184 ceph-mon[23692]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:07 smithi184 ceph-mon[30019]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:07 smithi196 ceph-mon[25662]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:09 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:09.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:09 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:11.375 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-09-20T19:47:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:11 smithi184 ceph-mon[23692]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:11.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:11 smithi184 ceph-mon[30019]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:11 smithi196 ceph-mon[25662]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:11.992 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T19:47:11.994 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T19:47:11.995 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T19:47:12.196 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:12.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:12.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:12 smithi196 ceph-mon[25662]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[30019]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:12 smithi184 ceph-mon[23692]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T19:47:13.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[30019]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-20T19:47:13.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:13.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:47:13.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T19:47:13.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:13.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:13 smithi184 ceph-mon[23692]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi196 to 4016M 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: from='osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-mon[25662]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-20T19:47:13.641 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[42236]: 2024-09-20T19:47:13.400+0000 7f0e000c8640 -1 osd.7 0 waiting for initial osdmap 2024-09-20T19:47:13.641 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 19:47:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[42236]: 2024-09-20T19:47:13.409+0000 7f0dfbef1640 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:47:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:14.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:47:14.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T19:47:14.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657] boot 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/809793211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[30019]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657] boot 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T19:47:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/809793211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:47:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:14 smithi184 ceph-mon[23692]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1726861633,"num_in_osds":8,"osd_in_since":1726861616,"num_remapped_pgs":1} 2024-09-20T19:47:14.816 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: osd.7 [v2:172.21.15.196:6824/3210812657,v1:172.21.15.196:6825/3210812657] boot 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/809793211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:47:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:14 smithi196 ceph-mon[25662]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-20T19:47:15.010 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[23692]: purged_snaps scrub starts 2024-09-20T19:47:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[23692]: purged_snaps scrub ok 2024-09-20T19:47:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[23692]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T19:47:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[30019]: purged_snaps scrub starts 2024-09-20T19:47:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[30019]: purged_snaps scrub ok 2024-09-20T19:47:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:15 smithi184 ceph-mon[30019]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T19:47:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:15 smithi196 ceph-mon[25662]: purged_snaps scrub starts 2024-09-20T19:47:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:15 smithi196 ceph-mon[25662]: purged_snaps scrub ok 2024-09-20T19:47:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:15 smithi196 ceph-mon[25662]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T19:47:16.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:16 smithi196 ceph-mon[25662]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T19:47:16.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:16 smithi196 ceph-mon[25662]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:16 smithi184 ceph-mon[23692]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T19:47:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:16 smithi184 ceph-mon[23692]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:16 smithi184 ceph-mon[30019]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T19:47:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:16 smithi184 ceph-mon[30019]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:16.831 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":58,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","created":"2024-09-20T19:42:18.687355+0000","modified":"2024-09-20T19:47:16.517175+0000","last_up_change":"2024-09-20T19:47:13.498635+0000","last_in_change":"2024-09-20T19:46:56.315667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:45:22.316751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18242be3-3630-4599-8612-ba05cd943947","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6803","nonce":3004532610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6805","nonce":3004532610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6809","nonce":3004532610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6807","nonce":3004532610}]},"public_addr":"172.21.15.184:6803/3004532610","cluster_addr":"172.21.15.184:6805/3004532610","heartbeat_back_addr":"172.21.15.184:6809/3004532610","heartbeat_front_addr":"172.21.15.184:6807/3004532610","state":["exists","up"]},{"osd":1,"uuid":"c5d9b3e3-baf4-4c10-bec1-99c167584ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6811","nonce":3361797109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6813","nonce":3361797109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6817","nonce":3361797109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6815","nonce":3361797109}]},"public_addr":"172.21.15.184:6811/3361797109","cluster_addr":"172.21.15.184:6813/3361797109","heartbeat_back_addr":"172.21.15.184:6817/3361797109","heartbeat_front_addr":"172.21.15.184:6815/3361797109","state":["exists","up"]},{"osd":2,"uuid":"0775d00f-951a-431d-9686-6531ffa73c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6819","nonce":795619898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6821","nonce":795619898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6825","nonce":795619898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6823","nonce":795619898}]},"public_addr":"172.21.15.184:6819/795619898","cluster_addr":"172.21.15.184:6821/795619898","heartbeat_back_addr":"172.21.15.184:6825/795619898","heartbeat_front_addr":"172.21.15.184:6823/795619898","state":["exists","up"]},{"osd":3,"uuid":"9ca7dee9-4973-46bb-bd7e-93f451d719c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6827","nonce":3829952175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6829","nonce":3829952175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6833","nonce":3829952175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6831","nonce":3829952175}]},"public_addr":"172.21.15.184:6827/3829952175","cluster_addr":"172.21.15.184:6829/3829952175","heartbeat_back_addr":"172.21.15.184:6833/3829952175","heartbeat_front_addr":"172.21.15.184:6831/3829952175","state":["exists","up"]},{"osd":4,"uuid":"f8a1cfe5-33b4-4383-8ada-32ef13d3b11c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6801","nonce":3871353859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6803","nonce":3871353859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6807","nonce":3871353859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6805","nonce":3871353859}]},"public_addr":"172.21.15.196:6801/3871353859","cluster_addr":"172.21.15.196:6803/3871353859","heartbeat_back_addr":"172.21.15.196:6807/3871353859","heartbeat_front_addr":"172.21.15.196:6805/3871353859","state":["exists","up"]},{"osd":5,"uuid":"5943d045-1e02-4e0e-a0d5-0ba436281904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6809","nonce":3149907518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6811","nonce":3149907518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6815","nonce":3149907518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6813","nonce":3149907518}]},"public_addr":"172.21.15.196:6809/3149907518","cluster_addr":"172.21.15.196:6811/3149907518","heartbeat_back_addr":"172.21.15.196:6815/3149907518","heartbeat_front_addr":"172.21.15.196:6813/3149907518","state":["exists","up"]},{"osd":6,"uuid":"7844d708-111d-4284-b5f7-017b77a8cc4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6817","nonce":895991492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6819","nonce":895991492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6823","nonce":895991492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6821","nonce":895991492}]},"public_addr":"172.21.15.196:6817/895991492","cluster_addr":"172.21.15.196:6819/895991492","heartbeat_back_addr":"172.21.15.196:6823/895991492","heartbeat_front_addr":"172.21.15.196:6821/895991492","state":["exists","up"]},{"osd":7,"uuid":"7918d00a-88b7-4a38-8442-9d2e9cc40e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6825","nonce":3210812657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6827","nonce":3210812657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6831","nonce":3210812657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6829","nonce":3210812657}]},"public_addr":"172.21.15.196:6825/3210812657","cluster_addr":"172.21.15.196:6827/3210812657","heartbeat_back_addr":"172.21.15.196:6831/3210812657","heartbeat_front_addr":"172.21.15.196:6829/3210812657","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:32.009854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:54.193608+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:18.687570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:42.512679+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:04.881285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:27.488029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:50.048989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:47:12.772104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6801/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/1313825268":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/3680121705":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/1216562938":"2024-09-21T19:42:42.511485+0000","172.21.15.184:6801/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/1689245221":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/375641286":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/4071369379":"2024-09-21T19:42:42.511485+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:47:17.521 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T19:45:22.316751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T19:47:17.522 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T19:47:17.717 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[23692]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2274315005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[30019]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T19:47:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2274315005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:17 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:17 smithi196 ceph-mon[25662]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T19:47:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2274315005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:17 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:18 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi196 2024-09-20T19:47:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:18 smithi196 ceph-mon[25662]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T19:47:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:18 smithi196 ceph-mon[25662]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi196 2024-09-20T19:47:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[23692]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T19:47:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[23692]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi196 2024-09-20T19:47:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[30019]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T19:47:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:18 smithi184 ceph-mon[30019]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:19.439 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 1 2024-09-20T19:47:19.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:19 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3612162245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T19:47:19.741 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:19 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3612162245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T19:47:19.742 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:19 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3612162245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T19:47:20.188 INFO:tasks.cephadm:Adding prometheus.a on smithi196 2024-09-20T19:47:20.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi196=a' 2024-09-20T19:47:20.396 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:47:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:20 smithi184 ceph-mon[23692]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T19:47:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:20 smithi184 ceph-mon[30019]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T19:47:20.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:20 smithi196 ceph-mon[25662]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T19:47:22.201 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled prometheus update... 2024-09-20T19:47:22.888 DEBUG:teuthology.orchestra.run.smithi196:prometheus.a> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T19:47:22.890 INFO:tasks.cephadm:Adding node-exporter.a on smithi184 2024-09-20T19:47:22.890 INFO:tasks.cephadm:Adding node-exporter.b on smithi196 2024-09-20T19:47:22.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi184=a;smithi196=b' 2024-09-20T19:47:23.149 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:47:23.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:23.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:23 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: Saving service prometheus spec with placement smithi196=a;count:1 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:23 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[23692]: Deploying daemon prometheus.a on smithi196 2024-09-20T19:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[30019]: Deploying daemon prometheus.a on smithi196 2024-09-20T19:47:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:24 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:24 smithi196 ceph-mon[25662]: Deploying daemon prometheus.a on smithi196 2024-09-20T19:47:24.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:24.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:24 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:25.342 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled node-exporter update... 2024-09-20T19:47:25.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:25 smithi196 ceph-mon[25662]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:47:25.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:25 smithi184 ceph-mon[23692]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:47:25.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:25 smithi184 ceph-mon[30019]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-20T19:47:26.304 DEBUG:teuthology.orchestra.run.smithi184:node-exporter.a> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T19:47:26.307 DEBUG:teuthology.orchestra.run.smithi196:node-exporter.b> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T19:47:26.309 INFO:tasks.cephadm:Adding alertmanager.a on smithi184 2024-09-20T19:47:26.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi184=a' 2024-09-20T19:47:26.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[23692]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi184=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[23692]: Saving service node-exporter spec with placement smithi184=a;smithi196=b;count:2 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[23692]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[30019]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi184=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[30019]: Saving service node-exporter spec with placement smithi184=a;smithi196=b;count:2 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:26.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:26 smithi184 ceph-mon[30019]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-20T19:47:26.540 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:26 smithi196 ceph-mon[25662]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi184=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:26 smithi196 ceph-mon[25662]: Saving service node-exporter spec with placement smithi184=a;smithi196=b;count:2 2024-09-20T19:47:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:26 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:26 smithi196 ceph-mon[25662]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-20T19:47:29.101 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled alertmanager update... 2024-09-20T19:47:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:29 smithi184 ceph-mon[23692]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-20T19:47:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:29 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:29 smithi184 ceph-mon[30019]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-20T19:47:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:29 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:29 smithi196 ceph-mon[25662]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-20T19:47:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:29 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:30 smithi184 ceph-mon[23692]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:30 smithi184 ceph-mon[23692]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-09-20T19:47:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:30 smithi184 ceph-mon[30019]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:30 smithi184 ceph-mon[30019]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-09-20T19:47:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:30 smithi196 ceph-mon[25662]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:30 smithi196 ceph-mon[25662]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-09-20T19:47:30.980 DEBUG:teuthology.orchestra.run.smithi184:alertmanager.a> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T19:47:30.983 INFO:tasks.cephadm:Adding grafana.a on smithi196 2024-09-20T19:47:30.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi196=a' 2024-09-20T19:47:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:31 smithi184 ceph-mon[23692]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-20T19:47:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:31 smithi184 ceph-mon[30019]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-20T19:47:31.502 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:47:31.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:31 smithi196 ceph-mon[25662]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-20T19:47:33.356 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:33 smithi196 ceph-mon[25662]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:33 smithi184 ceph-mon[23692]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:33 smithi184 ceph-mon[30019]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:35 smithi184 ceph-mon[23692]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:35 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:35 smithi184 ceph-mon[30019]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:35 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:35 smithi196 ceph-mon[25662]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:35 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:35.669 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled grafana update... 2024-09-20T19:47:36.253 DEBUG:teuthology.orchestra.run.smithi196:grafana.a> sudo journalctl -f -n 0 -u ceph-4b598dd2-7788-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T19:47:36.255 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T19:47:36.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:47:36.466 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:36 smithi184 ceph-mon[23692]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:36 smithi184 ceph-mon[23692]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-20T19:47:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:36 smithi184 ceph-mon[30019]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:36 smithi184 ceph-mon[30019]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-20T19:47:36.558 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:36 smithi196 ceph-mon[25662]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:47:36.558 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:36 smithi196 ceph-mon[25662]: Saving service grafana spec with placement smithi196=a;count:1 2024-09-20T19:47:37.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:37 smithi196 ceph-mon[25662]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:37 smithi184 ceph-mon[23692]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:37 smithi184 ceph-mon[30019]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:38.388 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 podman[44748]: 2024-09-20 19:47:38.362947762 +0000 UTC m=+0.354939984 container start f4b01811eefce5f7419ab947bf53e62588ddba77956815377fc79ce754bee253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:47:38.398 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2024-09-20T19:47:38.398 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBa0e1me/JnFxAA1e99uqgNsKqF2XbEhQFlAA== 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3642461954' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.386Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.386Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.387Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.387Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.387Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.388Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.388Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.389Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.389Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.391Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.392Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.57µs 2024-09-20T19:47:38.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.392Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.392Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.392Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.13µs wal_replay_duration=523.257µs wbl_replay_duration=118ns total_replay_duration=614.528µs 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.393Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.393Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.393Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.413Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.12429ms db_storage=879ns remote_storage=1.191µs web_handler=377ns query_engine=622ns scrape=472.033µs scrape_sd=88.993µs notify=890ns notify_sd=2.459µs rules=19.257697ms tracing=7.083µs 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.413Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:47:38.413Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 bash[44748]: f4b01811eefce5f7419ab947bf53e62588ddba77956815377fc79ce754bee253 2024-09-20T19:47:38.642 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:47:38 smithi196 systemd[1]: Started Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:47:38.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3642461954' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3642461954' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' 2024-09-20T19:47:38.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:38 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T19:47:39.065 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-20T19:47:39.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T19:47:39.065 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T19:47:39.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:47:39.588 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: ignoring --setuser ceph since I am not root 2024-09-20T19:47:39.588 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: ignoring --setgroup ceph since I am not root 2024-09-20T19:47:39.882 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setuser ceph since I am not root 2024-09-20T19:47:39.883 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setgroup ceph since I am not root 2024-09-20T19:47:39.883 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:39.677+0000 7f6b83d63040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:47:39.883 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:39.733+0000 7f6b83d63040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:47:39.883 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:39.880+0000 7f6b83d63040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:47:39.889 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-mgr[27887]: -- 172.21.15.196:0/480091130 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ff5904d00 con 0x563ff58d9000 2024-09-20T19:47:39.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:39.686+0000 7fb7f9da5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:47:39.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:39.744+0000 7fb7f9da5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:47:40.212 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:39.900+0000 7fb7f9da5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:47:40.212 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:39.970+0000 7fb7f9da5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:47:40.212 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:40.121+0000 7fb7f9da5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:47:40.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:39.945+0000 7f6b83d63040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:47:40.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:40.091+0000 7f6b83d63040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:47:40.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:40.229+0000 7f6b83d63040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:47:40.534 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:40.266+0000 7fb7f9da5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:47:40.593 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T19:47:40.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-mon[23692]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:47:40.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-mon[23692]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:47:40.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-mon[30019]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:47:40.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-mon[30019]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:47:40.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:40.532+0000 7f6b83d63040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:47:40.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:40.595+0000 7f6b83d63040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:47:40.787 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:40.784+0000 7f6b83d63040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:47:40.844 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-mon[25662]: from='mgr.14150 172.21.15.184:0/2106716638' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:47:40.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-mon[25662]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T19:47:40.845 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:40.609+0000 7fb7f9da5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:47:40.845 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:40.675+0000 7fb7f9da5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:47:41.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:40.863+0000 7fb7f9da5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:47:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:41.705+0000 7f6b83d63040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:47:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:41.774+0000 7f6b83d63040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:47:42.061 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:41.760+0000 7fb7f9da5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:47:42.061 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:41.832+0000 7fb7f9da5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:47:42.532 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-09-20T19:47:42.532 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQBe0e1mit5vHxAAT233YKExJz7Xm/UNzZ7Vag== 2024-09-20T19:47:42.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[30019]: from='client.? 172.21.15.196:0/180366042' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:42.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[23692]: from='client.? 172.21.15.196:0/180366042' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:42.740 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:42.481+0000 7f6b83d63040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:47:42.740 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:42.599+0000 7f6b83d63040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:47:42.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-mon[25662]: from='client.? 172.21.15.196:0/180366042' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:47:42.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:47:42.794 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:42.584+0000 7fb7f9da5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:47:42.794 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:42.712+0000 7fb7f9da5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:47:43.012 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:42.737+0000 7f6b83d63040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:47:43.012 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:42.798+0000 7f6b83d63040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:47:43.012 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:42.944+0000 7f6b83d63040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:47:43.067 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:42.857+0000 7fb7f9da5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:47:43.067 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:42 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:42.917+0000 7fb7f9da5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:47:43.067 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.065+0000 7fb7f9da5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:47:43.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.010+0000 7f6b83d63040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:47:43.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.069+0000 7f6b83d63040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:47:43.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.223+0000 7f6b83d63040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:47:43.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.288+0000 7f6b83d63040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:47:43.351 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.135+0000 7fb7f9da5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:47:43.351 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.194+0000 7fb7f9da5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:47:43.385 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-20T19:47:43.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T19:47:43.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T19:47:43.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-20T19:47:43.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.354+0000 7f6b83d63040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:47:43.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.436+0000 7f6b83d63040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:47:43.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.499+0000 7f6b83d63040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:47:43.628 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:43.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.350+0000 7fb7f9da5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:47:43.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.421+0000 7fb7f9da5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:47:43.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.488+0000 7fb7f9da5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:47:43.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.573+0000 7fb7f9da5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:47:43.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.631+0000 7fb7f9da5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:47:43.869 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:43.562+0000 7f6b83d63040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:47:43.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:43.690+0000 7fb7f9da5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:47:44.438 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:44.375+0000 7f6b83d63040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:47:44] ENGINE Bus STARTING 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: CherryPy Checker: 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: The Application mounted at '' has an empty config. 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:47:44] ENGINE Serving on http://:::9283 2024-09-20T19:47:44.765 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:47:44] ENGINE Bus STARTED 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: Active manager daemon y restarted 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: Activating manager daemon y 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: mgrmap e17: y(active, starting, since 0.00672585s), standbys: x 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:47:44.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: Manager daemon y is now available 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:44.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: Active manager daemon y restarted 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: Activating manager daemon y 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: mgrmap e17: y(active, starting, since 0.00672585s), standbys: x 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:47:44.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: Manager daemon y is now available 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:44.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:44.792 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:44.479+0000 7fb7f9da5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: [20/Sep/2024:19:47:44] ENGINE Bus STARTING 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: CherryPy Checker: 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: The Application mounted at '' has an empty config. 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: [20/Sep/2024:19:47:44] ENGINE Serving on http://:::9283 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[27883]: [20/Sep/2024:19:47:44] ENGINE Bus STARTED 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: Active manager daemon y restarted 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: Activating manager daemon y 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: mgrmap e17: y(active, starting, since 0.00672585s), standbys: x 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:47:44.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:47:44.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:47:44.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: Manager daemon y is now available 2024-09-20T19:47:44.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:44.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:47:44.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: Standby manager daemon x restarted 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: Standby manager daemon x started 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:47:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[23692]: mgrmap e18: y(active, since 1.01241s), standbys: x 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:47:45.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: Standby manager daemon x restarted 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:47:45.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-mon[30019]: mgrmap e18: y(active, since 1.01241s), standbys: x 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:47:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: Standby manager daemon x restarted 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.196:0/929851091' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:47:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:45 smithi196 ceph-mon[25662]: mgrmap e18: y(active, since 1.01241s), standbys: x 2024-09-20T19:47:46.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:45.953+0000 7f6ae19dd640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:47:45] ENGINE Error in HTTPServer.serve 2024-09-20T19:47:46.062 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: Traceback (most recent call last): 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._connections.run(self.expiration_interval) 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._run(expiration_interval) 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: s = self.context.wrap_socket( 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:47:46.063 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: return self.sslsocket_class._create( 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self.do_handshake() 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: self._sslobj.do_handshake() 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:47:46.064 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:47:46.537 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:47:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:[{"version":24,"timestamp":"2024-09-20T19:47:44.430360+0000","name":"","changes":[{"name":"osd/host:smithi184/osd_memory_target","previous_value":"3942680268"}]}] 2024-09-20T19:47:47.621 INFO:tasks.ceph_manager:config epoch is 24 2024-09-20T19:47:47.621 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T19:47:47.621 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T19:47:47.621 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T19:47:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: Deploying daemon alertmanager.a on smithi184 2024-09-20T19:47:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: [20/Sep/2024:19:47:45] ENGINE Bus STARTING 2024-09-20T19:47:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: [20/Sep/2024:19:47:45] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: [20/Sep/2024:19:47:45] ENGINE Error in HTTPServer.serve 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:47:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: [20/Sep/2024:19:47:46] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: [20/Sep/2024:19:47:46] ENGINE Bus STARTED 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1692749278' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[23692]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: Deploying daemon alertmanager.a on smithi184 2024-09-20T19:47:47.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: [20/Sep/2024:19:47:45] ENGINE Bus STARTING 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: [20/Sep/2024:19:47:45] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: [20/Sep/2024:19:47:45] ENGINE Error in HTTPServer.serve 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:47:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.do_handshake() 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: [20/Sep/2024:19:47:46] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: [20/Sep/2024:19:47:46] ENGINE Bus STARTED 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1692749278' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T19:47:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:47 smithi184 ceph-mon[30019]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: Deploying daemon alertmanager.a on smithi184 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: [20/Sep/2024:19:47:45] ENGINE Bus STARTING 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: [20/Sep/2024:19:47:45] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: [20/Sep/2024:19:47:45] ENGINE Error in HTTPServer.serve 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:47:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: [20/Sep/2024:19:47:46] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: [20/Sep/2024:19:47:46] ENGINE Bus STARTED 2024-09-20T19:47:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:47.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1692749278' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T19:47:47.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:47 smithi196 ceph-mon[25662]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T19:47:48.459 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:49 smithi184 ceph-mon[23692]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:49 smithi184 ceph-mon[23692]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T19:47:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:49 smithi184 ceph-mon[30019]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:49 smithi184 ceph-mon[30019]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T19:47:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:49 smithi196 ceph-mon[25662]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:47:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:49 smithi196 ceph-mon[25662]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T19:47:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:51 smithi184 ceph-mon[23692]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T19:47:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:51 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1557707923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T19:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:51 smithi184 ceph-mon[30019]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T19:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:51 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1557707923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T19:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:51 smithi196 ceph-mon[25662]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T19:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:51 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1557707923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T19:47:52.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:47:52] "GET /metrics HTTP/1.1" 200 23685 "" "Prometheus/2.43.0" 2024-09-20T19:47:52.309 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":20,"flags":0,"active_gid":24338,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1203091601},{"type":"v1","addr":"172.21.15.184:6801","nonce":1203091601}]},"active_addr":"172.21.15.184:6801/1203091601","active_change":"2024-09-20T19:47:44.377871+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24343,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.184:8443/","prometheus":"http://172.21.15.184:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":60,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":265488425}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":3901329294}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":129943563}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":125266441}]}]} 2024-09-20T19:47:52.315 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T19:47:52.315 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T19:47:52.316 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:47:52.513 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:53.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:53 smithi184 ceph-mon[23692]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:47:53.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:53 smithi184 ceph-mon[30019]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:47:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:53 smithi196 ceph-mon[25662]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:47:55.277 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:55.277 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":60,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","created":"2024-09-20T19:42:18.687355+0000","modified":"2024-09-20T19:47:44.377755+0000","last_up_change":"2024-09-20T19:47:13.498635+0000","last_in_change":"2024-09-20T19:46:56.315667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:45:22.316751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18242be3-3630-4599-8612-ba05cd943947","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6803","nonce":3004532610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6805","nonce":3004532610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6809","nonce":3004532610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6807","nonce":3004532610}]},"public_addr":"172.21.15.184:6803/3004532610","cluster_addr":"172.21.15.184:6805/3004532610","heartbeat_back_addr":"172.21.15.184:6809/3004532610","heartbeat_front_addr":"172.21.15.184:6807/3004532610","state":["exists","up"]},{"osd":1,"uuid":"c5d9b3e3-baf4-4c10-bec1-99c167584ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6811","nonce":3361797109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6813","nonce":3361797109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6817","nonce":3361797109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6815","nonce":3361797109}]},"public_addr":"172.21.15.184:6811/3361797109","cluster_addr":"172.21.15.184:6813/3361797109","heartbeat_back_addr":"172.21.15.184:6817/3361797109","heartbeat_front_addr":"172.21.15.184:6815/3361797109","state":["exists","up"]},{"osd":2,"uuid":"0775d00f-951a-431d-9686-6531ffa73c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6819","nonce":795619898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6821","nonce":795619898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6825","nonce":795619898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6823","nonce":795619898}]},"public_addr":"172.21.15.184:6819/795619898","cluster_addr":"172.21.15.184:6821/795619898","heartbeat_back_addr":"172.21.15.184:6825/795619898","heartbeat_front_addr":"172.21.15.184:6823/795619898","state":["exists","up"]},{"osd":3,"uuid":"9ca7dee9-4973-46bb-bd7e-93f451d719c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6827","nonce":3829952175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6829","nonce":3829952175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6833","nonce":3829952175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6831","nonce":3829952175}]},"public_addr":"172.21.15.184:6827/3829952175","cluster_addr":"172.21.15.184:6829/3829952175","heartbeat_back_addr":"172.21.15.184:6833/3829952175","heartbeat_front_addr":"172.21.15.184:6831/3829952175","state":["exists","up"]},{"osd":4,"uuid":"f8a1cfe5-33b4-4383-8ada-32ef13d3b11c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6801","nonce":3871353859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6803","nonce":3871353859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6807","nonce":3871353859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6805","nonce":3871353859}]},"public_addr":"172.21.15.196:6801/3871353859","cluster_addr":"172.21.15.196:6803/3871353859","heartbeat_back_addr":"172.21.15.196:6807/3871353859","heartbeat_front_addr":"172.21.15.196:6805/3871353859","state":["exists","up"]},{"osd":5,"uuid":"5943d045-1e02-4e0e-a0d5-0ba436281904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6809","nonce":3149907518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6811","nonce":3149907518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6815","nonce":3149907518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6813","nonce":3149907518}]},"public_addr":"172.21.15.196:6809/3149907518","cluster_addr":"172.21.15.196:6811/3149907518","heartbeat_back_addr":"172.21.15.196:6815/3149907518","heartbeat_front_addr":"172.21.15.196:6813/3149907518","state":["exists","up"]},{"osd":6,"uuid":"7844d708-111d-4284-b5f7-017b77a8cc4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6817","nonce":895991492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6819","nonce":895991492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6823","nonce":895991492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6821","nonce":895991492}]},"public_addr":"172.21.15.196:6817/895991492","cluster_addr":"172.21.15.196:6819/895991492","heartbeat_back_addr":"172.21.15.196:6823/895991492","heartbeat_front_addr":"172.21.15.196:6821/895991492","state":["exists","up"]},{"osd":7,"uuid":"7918d00a-88b7-4a38-8442-9d2e9cc40e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6825","nonce":3210812657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6827","nonce":3210812657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6831","nonce":3210812657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6829","nonce":3210812657}]},"public_addr":"172.21.15.196:6825/3210812657","cluster_addr":"172.21.15.196:6827/3210812657","heartbeat_back_addr":"172.21.15.196:6831/3210812657","heartbeat_front_addr":"172.21.15.196:6829/3210812657","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:32.009854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:54.193608+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:18.687570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:42.512679+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:04.881285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:27.488029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:50.048989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:47:12.772104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6801/2073022965":"2024-09-21T19:47:44.377722+0000","172.21.15.184:6800/2073022965":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/2187783388":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/4071369379":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/375641286":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/1689245221":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/1216562938":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/3680121705":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/1313825268":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/3685186450":"2024-09-21T19:47:44.377722+0000","172.21.15.184:6801/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6801/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/3643808076":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/1486363706":"2024-09-21T19:47:44.377722+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:47:55.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:55 smithi184 ceph-mon[23692]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T19:47:55.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:55 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/59601790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:55.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:55 smithi184 ceph-mon[30019]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T19:47:55.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:55 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/59601790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:55 smithi196 ceph-mon[25662]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T19:47:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:55 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/59601790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:47:56.746 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T19:47:56.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:47:57.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:57 smithi184 ceph-mon[23692]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:47:57.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:57 smithi184 ceph-mon[30019]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:47:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:57 smithi196 ceph-mon[25662]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:47:57.820 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:59 smithi184 ceph-mon[23692]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:47:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:47:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:59 smithi184 ceph-mon[30019]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:47:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:47:59.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:47:59 smithi184 podman[50509]: 2024-09-20 19:47:59.497921662 +0000 UTC m=+0.222695854 volume create b3dc3145bec6fa0fce441d09168f13eb6dc6b259c080ac08d929e169bda40af3 2024-09-20T19:47:59.680 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:47:59.680 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":60,"fsid":"4b598dd2-7788-11ef-baf5-efdc52797490","created":"2024-09-20T19:42:18.687355+0000","modified":"2024-09-20T19:47:44.377755+0000","last_up_change":"2024-09-20T19:47:13.498635+0000","last_in_change":"2024-09-20T19:46:56.315667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:45:22.316751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18242be3-3630-4599-8612-ba05cd943947","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6803","nonce":3004532610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6805","nonce":3004532610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6809","nonce":3004532610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3004532610},{"type":"v1","addr":"172.21.15.184:6807","nonce":3004532610}]},"public_addr":"172.21.15.184:6803/3004532610","cluster_addr":"172.21.15.184:6805/3004532610","heartbeat_back_addr":"172.21.15.184:6809/3004532610","heartbeat_front_addr":"172.21.15.184:6807/3004532610","state":["exists","up"]},{"osd":1,"uuid":"c5d9b3e3-baf4-4c10-bec1-99c167584ae3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6811","nonce":3361797109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6813","nonce":3361797109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6817","nonce":3361797109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":3361797109},{"type":"v1","addr":"172.21.15.184:6815","nonce":3361797109}]},"public_addr":"172.21.15.184:6811/3361797109","cluster_addr":"172.21.15.184:6813/3361797109","heartbeat_back_addr":"172.21.15.184:6817/3361797109","heartbeat_front_addr":"172.21.15.184:6815/3361797109","state":["exists","up"]},{"osd":2,"uuid":"0775d00f-951a-431d-9686-6531ffa73c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6819","nonce":795619898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6821","nonce":795619898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6825","nonce":795619898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":795619898},{"type":"v1","addr":"172.21.15.184:6823","nonce":795619898}]},"public_addr":"172.21.15.184:6819/795619898","cluster_addr":"172.21.15.184:6821/795619898","heartbeat_back_addr":"172.21.15.184:6825/795619898","heartbeat_front_addr":"172.21.15.184:6823/795619898","state":["exists","up"]},{"osd":3,"uuid":"9ca7dee9-4973-46bb-bd7e-93f451d719c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6827","nonce":3829952175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6829","nonce":3829952175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6833","nonce":3829952175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":3829952175},{"type":"v1","addr":"172.21.15.184:6831","nonce":3829952175}]},"public_addr":"172.21.15.184:6827/3829952175","cluster_addr":"172.21.15.184:6829/3829952175","heartbeat_back_addr":"172.21.15.184:6833/3829952175","heartbeat_front_addr":"172.21.15.184:6831/3829952175","state":["exists","up"]},{"osd":4,"uuid":"f8a1cfe5-33b4-4383-8ada-32ef13d3b11c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6801","nonce":3871353859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6803","nonce":3871353859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6807","nonce":3871353859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3871353859},{"type":"v1","addr":"172.21.15.196:6805","nonce":3871353859}]},"public_addr":"172.21.15.196:6801/3871353859","cluster_addr":"172.21.15.196:6803/3871353859","heartbeat_back_addr":"172.21.15.196:6807/3871353859","heartbeat_front_addr":"172.21.15.196:6805/3871353859","state":["exists","up"]},{"osd":5,"uuid":"5943d045-1e02-4e0e-a0d5-0ba436281904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6809","nonce":3149907518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6811","nonce":3149907518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6815","nonce":3149907518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3149907518},{"type":"v1","addr":"172.21.15.196:6813","nonce":3149907518}]},"public_addr":"172.21.15.196:6809/3149907518","cluster_addr":"172.21.15.196:6811/3149907518","heartbeat_back_addr":"172.21.15.196:6815/3149907518","heartbeat_front_addr":"172.21.15.196:6813/3149907518","state":["exists","up"]},{"osd":6,"uuid":"7844d708-111d-4284-b5f7-017b77a8cc4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6817","nonce":895991492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6819","nonce":895991492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6823","nonce":895991492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":895991492},{"type":"v1","addr":"172.21.15.196:6821","nonce":895991492}]},"public_addr":"172.21.15.196:6817/895991492","cluster_addr":"172.21.15.196:6819/895991492","heartbeat_back_addr":"172.21.15.196:6823/895991492","heartbeat_front_addr":"172.21.15.196:6821/895991492","state":["exists","up"]},{"osd":7,"uuid":"7918d00a-88b7-4a38-8442-9d2e9cc40e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6825","nonce":3210812657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6827","nonce":3210812657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6831","nonce":3210812657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3210812657},{"type":"v1","addr":"172.21.15.196:6829","nonce":3210812657}]},"public_addr":"172.21.15.196:6825/3210812657","cluster_addr":"172.21.15.196:6827/3210812657","heartbeat_back_addr":"172.21.15.196:6831/3210812657","heartbeat_front_addr":"172.21.15.196:6829/3210812657","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:32.009854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:44:54.193608+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:18.687570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:45:42.512679+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:04.881285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:27.488029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:46:50.048989+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:47:12.772104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6801/2073022965":"2024-09-21T19:47:44.377722+0000","172.21.15.184:6800/2073022965":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/2187783388":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/4071369379":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/375641286":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/1689245221":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/1216562938":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/3680121705":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6800/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/1313825268":"2024-09-21T19:43:08.001585+0000","172.21.15.184:0/3685186450":"2024-09-21T19:47:44.377722+0000","172.21.15.184:6801/3616549371":"2024-09-21T19:43:08.001585+0000","172.21.15.184:6801/930113251":"2024-09-21T19:42:42.511485+0000","172.21.15.184:0/3643808076":"2024-09-21T19:47:44.377722+0000","172.21.15.184:0/1486363706":"2024-09-21T19:47:44.377722+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:47:59.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:59 smithi196 ceph-mon[25662]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:47:59.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:47:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:00.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:47:59 smithi184 podman[50509]: 2024-09-20 19:47:59.715880797 +0000 UTC m=+0.440654989 container create 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 podman[50509]: 2024-09-20 19:48:00.269382874 +0000 UTC m=+0.994157073 container init 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 podman[50509]: 2024-09-20 19:48:00.272411487 +0000 UTC m=+0.997185691 container start 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.285Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.286Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.287Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.184 port=9094 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.288Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T19:48:00.313 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.312Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:48:00.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:00] ENGINE Bus STOPPING 2024-09-20T19:48:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3819199757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:48:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3819199757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:48:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.312Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:48:00.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.315Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T19:48:00.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:00.315Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T19:48:00.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 bash[50509]: 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 2024-09-20T19:48:00.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:00 smithi184 systemd[1]: Started Ceph alertmanager.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:48:00.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T19:48:00.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T19:48:00.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T19:48:00.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T19:48:00.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T19:48:00.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T19:48:00.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T19:48:00.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T19:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:00 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3819199757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:00 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:00 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:00 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:00 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:48:01.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:01] ENGINE Bus STOPPED 2024-09-20T19:48:01.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:01] ENGINE Bus STARTING 2024-09-20T19:48:01.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:01] ENGINE Serving on http://:::9283 2024-09-20T19:48:01.294 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.294 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.294 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.295 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.295 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.301 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:01.429 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:48:01] ENGINE Bus STARTED 2024-09-20T19:48:01.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:48:01.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:48:01.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[30019]: Deploying daemon grafana.a on smithi196 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:01 smithi184 ceph-mon[23692]: Deploying daemon grafana.a on smithi196 2024-09-20T19:48:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:01 smithi196 ceph-mon[25662]: Deploying daemon grafana.a on smithi196 2024-09-20T19:48:02.289 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:02] "GET /metrics HTTP/1.1" 200 26214 "" "Prometheus/2.43.0" 2024-09-20T19:48:02.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:02.288Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000262376s 2024-09-20T19:48:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:03 smithi184 ceph-mon[23692]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:03 smithi184 ceph-mon[30019]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:03 smithi196 ceph-mon[25662]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:05.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:05 smithi184 ceph-mon[23692]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:05.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:05 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:05.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:05 smithi184 ceph-mon[30019]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:05.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:05 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:05 smithi196 ceph-mon[25662]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:05 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:06.826 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:06.826 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:07 smithi184 ceph-mon[23692]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:07 smithi184 ceph-mon[30019]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:07 smithi196 ceph-mon[25662]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:09.581 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:09 smithi184 ceph-mon[23692]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:09.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:09 smithi184 ceph-mon[30019]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:09.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:09 smithi196 ceph-mon[25662]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:10.463 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:48:10.290Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002432358s 2024-09-20T19:48:11.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:11 smithi196 ceph-mon[25662]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:11.765 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:11 smithi196 podman[46760]: 2024-09-20 19:48:11.519835451 +0000 UTC m=+0.015160619 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-20T19:48:11.765 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:11 smithi196 podman[46760]: 2024-09-20 19:48:11.764300626 +0000 UTC m=+0.259625832 container create 16f45b96cac73cb7c5fe6d9f5e08e6ce01bad60e8c07f7889f0421ae60ffc80a (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a, distribution-scope=public, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.32.0, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8.5, io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-09-20T19:48:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:11 smithi184 ceph-mon[23692]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:11 smithi184 ceph-mon[30019]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:12.025 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:11 smithi196 podman[46760]: 2024-09-20 19:48:11.958096318 +0000 UTC m=+0.453421495 container init 16f45b96cac73cb7c5fe6d9f5e08e6ce01bad60e8c07f7889f0421ae60ffc80a (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, maintainer=Paul Cuzner , release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, architecture=x86_64, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8) 2024-09-20T19:48:12.025 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:11 smithi196 podman[46760]: 2024-09-20 19:48:11.960739826 +0000 UTC m=+0.456064999 container start 16f45b96cac73cb7c5fe6d9f5e08e6ce01bad60e8c07f7889f0421ae60ffc80a (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, maintainer=Paul Cuzner , io.openshift.expose-services=, description=Ceph Grafana Container, version=8.5, vendor=Red Hat, Inc., distribution-scope=public) 2024-09-20T19:48:12.202 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:12] "GET /metrics HTTP/1.1" 200 26214 "" "Prometheus/2.43.0" 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 bash[46760]: 16f45b96cac73cb7c5fe6d9f5e08e6ce01bad60e8c07f7889f0421ae60ffc80a 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 systemd[1]: Started Ceph grafana.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115164845Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115339652Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115357913Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115364334Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115370024Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T19:48:12.278 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115376329Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115381928Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115387522Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115393216Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115398721Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115403871Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115408906Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115414251Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=settings t=2024-09-20T19:48:12.115419548Z level=info msg="App mode production" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore t=2024-09-20T19:48:12.115479663Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore t=2024-09-20T19:48:12.115502857Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.116371095Z level=info msg="Starting DB migrations" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.116443908Z level=info msg="Executing migration" id="create migration_log table" 2024-09-20T19:48:12.279 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.116978947Z level=info msg="Executing migration" id="create user table" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.117418873Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.11789523Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.118365354Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.118870921Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.11927451Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.119978126Z level=info msg="Executing migration" id="create user table v2" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.120375629Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.120745076Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.121174091Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.121501825Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.121857262Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.122288529Z level=info msg="Executing migration" id="Update user table charset" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.122563615Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-20T19:48:12.280 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.123024027Z level=info msg="Executing migration" id="Add missing user data" 2024-09-20T19:48:12.281 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.123391492Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-20T19:48:12.281 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.123798143Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-20T19:48:12.281 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.124207534Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-20T19:48:12.281 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.124619945Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.125721882Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.126221001Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.12666751Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.127082254Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.127484287Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.127877919Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.128177681Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.128530043Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.128917487Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-20T19:48:12.282 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.129300648Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.1296577Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.130340124Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.130710791Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.131129862Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.131495935Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.131854819Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.132250349Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.132540349Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.132847545Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.133175016Z level=info msg="Executing migration" id="create star table" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.133502141Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.133890353Z level=info msg="Executing migration" id="create org table v1" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.134269449Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-20T19:48:12.283 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.134643738Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.137337107Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.137737307Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.138150758Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.138551798Z level=info msg="Executing migration" id="Update org table charset" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.141237723Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.141558909Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.143996681Z level=info msg="Executing migration" id="create dashboard table" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.144401124Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.146732176Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.149568297Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.152236575Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.155093173Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.155814086Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T19:48:12.284 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.156659369Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.157491249Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.15828677Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.159086094Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.159908648Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.160903678Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.16160346Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.162536249Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.163254179Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.164062102Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.164863774Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.165729034Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.166508368Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.167293366Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-20T19:48:12.285 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.168144767Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.169085315Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.169814286Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.170330873Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.171162287Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.171680325Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.172056533Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.172890682Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.173759964Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.174585479Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.175538207Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.176331991Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.17725245Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-20T19:48:12.286 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.178276254Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.178968254Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.179703053Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.180691582Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.181422589Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.182383344Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.183115242Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.1840149Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.184951855Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.185663368Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.186712479Z level=info msg="Executing migration" id="create data_source table" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.187438893Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.188420624Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.189104533Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-20T19:48:12.287 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.189875004Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.190819562Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.192036702Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.19501347Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.197791943Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.20055274Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.20344897Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.206081395Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.208891058Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.21163604Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.214597058Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.217478367Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.220235934Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.223071071Z level=info msg="Executing migration" id="Add uid column" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.225915998Z level=info msg="Executing migration" id="Update uid value" 2024-09-20T19:48:12.288 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.228799004Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.231524736Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.234213231Z level=info msg="Executing migration" id="create api_key table" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.236962242Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.239874383Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.242709308Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.245282236Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.247972478Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.248503351Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.251210908Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.253892152Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.256502066Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.256983017Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.259807687Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.262354405Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-20T19:48:12.289 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.265178443Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.265736386Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.268449607Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.271189481Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.273575979Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.273931835Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.276679655Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.279478155Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.280290772Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-20T19:48:12.290 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.280807291Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-20T19:48:12.528 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.281580963Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.282423482Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.283231131Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.283649592Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.284482635Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.285330033Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.286322979Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.286992134Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.287961403Z level=info msg="Executing migration" id="create quota table v1" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.28872858Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.289522275Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.28980843Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.290215546Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.290629909Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-20T19:48:12.529 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.291472454Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-20T19:48:12.530 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.292353073Z level=info msg="Executing migration" id="create session table" 2024-09-20T19:48:12.530 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.292820135Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-20T19:48:12.530 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.293638151Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-20T19:48:12.530 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.294567873Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.295405511Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.295794109Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.296702679Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.297637929Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.298423082Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.298730465Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.299670184Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.300567742Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.301365495Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.301651839Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.302298797Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.302882203Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-20T19:48:12.531 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.303887551Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.304594058Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.305057256Z level=info msg="Executing migration" id="create alert table v1" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.307892008Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.310821463Z level=info msg="Executing migration" id="add index alert state" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.313745314Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.314327159Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.317173293Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.320021533Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.322858533Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.325529403Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.328273085Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.330944396Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-20T19:48:12.532 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.333833048Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.336818192Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.339393774Z level=info msg="Executing migration" id="Add column is_default" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.34224985Z level=info msg="Executing migration" id="Add column frequency" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.345053866Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.347929592Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.350686921Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.353407579Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.355855719Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.358274813Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.361085866Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.363840779Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.366268538Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.369031294Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.371791645Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-20T19:48:12.533 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.374431745Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.377129926Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.379710873Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.382480617Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.384983416Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.387626678Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.390297257Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.393083391Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.395863737Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.396590598Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.397529203Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.398259986Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.399119602Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.399921175Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-20T19:48:12.534 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.400815295Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.401560694Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.402515389Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.403255939Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.404095406Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.40494441Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.405909649Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.40659274Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.40744907Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.409014786Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.409888942Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.410835009Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.411594873Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.41459135Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.417420292Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-20T19:48:12.535 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.420456113Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.423142791Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.426315364Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.429063098Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.431939905Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.434482531Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.437418752Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.440111079Z level=info msg="Executing migration" id="Move region to single row" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.442966331Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.445621711Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.448541612Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.451323507Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.454077101Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.45682155Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-20T19:48:12.536 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.459434052Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.46221762Z level=info msg="Executing migration" id="create test_data table" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.46489051Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.465330294Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.468085333Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.470833485Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.47334738Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.476041421Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.478669539Z level=info msg="Executing migration" id="create team table" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.481554128Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-20T19:48:12.537 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.484050851Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.486944688Z level=info msg="Executing migration" id="create team member table" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.489574064Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.492336931Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.494955588Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.497459087Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.500316299Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.502962167Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.505783067Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.508502431Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.509334058Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.510044207Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.510537157Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-20T19:48:12.538 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.511465385Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.512129555Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.512910105Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.513328982Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.514250102Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.514643559Z level=info msg="Executing migration" id="create tag table" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.515068979Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.515846919Z level=info msg="Executing migration" id="create login attempt table" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.516230348Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.517108566Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.517921506Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.519779515Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.52017349Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.521092532Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-20T19:48:12.539 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.521775144Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.522691416Z level=info msg="Executing migration" id="create user auth table" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.523308539Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.524338619Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.525049136Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.525884564Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.526635961Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.527481987Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.528372817Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.529214416Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.530044296Z level=info msg="Executing migration" id="create server_lock table" 2024-09-20T19:48:12.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.530886392Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-20T19:48:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:167503724567 2024-09-20T19:48:12.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T19:48:12.786 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.531738936Z level=info msg="Executing migration" id="create user auth token table" 2024-09-20T19:48:12.786 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.532577628Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-20T19:48:12.786 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.53302568Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.533875458Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.534752414Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.535605056Z level=info msg="Executing migration" id="create cache_data table" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.536029122Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.538974289Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.541746316Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.544608817Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.544929068Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.54523961Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.545682367Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.546108194Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.54651694Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.546778494Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-20T19:48:12.787 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.547178346Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.547546264Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.547965495Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.548371872Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.54914363Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.549570448Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-20T19:48:12.788 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.549840411Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.55024672Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.550635643Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.551050136Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.551331211Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.551722572Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.552131162Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.552545207Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.552970639Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.553735437Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.554145421Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.554530955Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.557895379Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.561232347Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T19:48:12.789 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.561655219Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.562064768Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.562862291Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.563264663Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.563676666Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.564126622Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.564537159Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.564801404Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.56559333Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.566406475Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.567265366Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.567666314Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.568082735Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.5688836Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-20T19:48:12.790 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.569674182Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.570089286Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.57087667Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.571667707Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.571992008Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.572412256Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.575170034Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.575613932Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.575931048Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.578934524Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.581498905Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.584501424Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.585407485Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.586298747Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.587141254Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-20T19:48:12.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.587957362Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.588794634Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.58973317Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.590805873Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.591734586Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.592706486Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-20T19:48:12.792 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.593457271Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.594255725Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.595136985Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.595529762Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.596398087Z level=info msg="Executing migration" id="create alert_image table" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.597234352Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.598071314Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.598391267Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.599232435Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.600065865Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T19:48:12.793 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.600159054Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.600961724Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.60131891Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.602130348Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.60225887Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.60236381Z level=info msg="alerts found to migrate" alerts=0 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.602742314Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.603184515Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.604026144Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.60443815Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.605201569Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.606066572Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.607002785Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.607556445Z level=info msg="Executing migration" id="create data_keys table" 2024-09-20T19:48:12.794 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.610512886Z level=info msg="Executing migration" id="create secrets table" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.613219746Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.617488727Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.618434682Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.621157278Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.625380121Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.629632813Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.630056888Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.63291587Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.635693846Z level=info msg="Executing migration" id="create permission table" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.638410731Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.638865059Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.641894147Z level=info msg="Executing migration" id="create role table" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.644555676Z level=info msg="Executing migration" id="add column display_name" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.647281767Z level=info msg="Executing migration" id="add column group_name" 2024-09-20T19:48:12.795 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.648343384Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.651269402Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.654000243Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.656655899Z level=info msg="Executing migration" id="create team role table" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.657071907Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.659875665Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.662521911Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.665363256Z level=info msg="Executing migration" id="create user role table" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.665922956Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.668702759Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.671407505Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.673833312Z level=info msg="Executing migration" id="create builtin role table" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.676532749Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.679192704Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-20T19:48:12.796 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.681933198Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.684615983Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.685195555Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.6877155Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.690606388Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.693151251Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.693565432Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.694544178Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.695684156Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.696556036Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.696997834Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.697954025Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.698683348Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.6996378Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-20T19:48:12.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.700144189Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.701090238Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.701803961Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.70277361Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.703252333Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.704212084Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.704991467Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.705983955Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.706524287Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.707534386Z level=info msg="Executing migration" id="add correlation config column" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.708643085Z level=info msg="Executing migration" id="create entity_events table" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.709653486Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.710249494Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.710327317Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:48:12.798 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.711131454Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.711227462Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.711928174Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.712812356Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.713262023Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.714078122Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.714850715Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.7158313Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.717502816Z level=info msg="Executing migration" id="Drop public config table" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.71820764Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.719066168Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.719915844Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.720398017Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.721231576Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.724368566Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-20T19:48:12.799 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.727282595Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.728316017Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.731188813Z level=info msg="Executing migration" id="add share column" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.734113825Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.736736594Z level=info msg="Executing migration" id="create file table" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.737255983Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.739967163Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.742746077Z level=info msg="Executing migration" id="create file_meta table" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.745538081Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.746035994Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.748811763Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.751748915Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.754619207Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-20T19:48:12.800 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.755238981Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.757981967Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.760669047Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.763482519Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.763992392Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.766561685Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-20T19:48:12.801 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.769196291Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.771839907Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.772850397Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.775688304Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.785272928Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.787816772Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-20T19:48:12.802 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.788431335Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-20T19:48:13.057 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.793822051Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-20T19:48:13.057 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.796490071Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-20T19:48:13.057 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.799133372Z level=info msg="Executing migration" id="create folder table" 2024-09-20T19:48:13.057 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.799538812Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-20T19:48:13.057 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.800398704Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.80121009Z level=info msg="Executing migration" id="Update folder title length" 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.802017771Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=migrator t=2024-09-20T19:48:12.802445759Z level=info msg="migrations completed" performed=485 skipped=0 duration=686.020849ms 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore t=2024-09-20T19:48:12.802849151Z level=info msg="Created default organization" 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=plugin.loader t=2024-09-20T19:48:12.862766219Z level=info msg="Plugin registered" pluginID=input 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=plugin.loader t=2024-09-20T19:48:12.891045662Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=plugin.loader t=2024-09-20T19:48:12.891094524Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=secrets t=2024-09-20T19:48:12.891134058Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=query_data t=2024-09-20T19:48:12.892841727Z level=info msg="Query Service initialization" 2024-09-20T19:48:13.058 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=live.push_http t=2024-09-20T19:48:12.895047482Z level=info msg="Live Push Gateway initialization" 2024-09-20T19:48:13.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: Deploying daemon node-exporter.a on smithi184 2024-09-20T19:48:13.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-mon[25662]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:13.381 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.381 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: Deploying daemon node-exporter.a on smithi184 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[23692]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: Deploying daemon node-exporter.a on smithi184 2024-09-20T19:48:13.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:13 smithi184 ceph-mon[30019]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:13.887 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=infra.usagestats.collector t=2024-09-20T19:48:13.688919601Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=provisioning.datasources t=2024-09-20T19:48:13.744924556Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=provisioning.datasources t=2024-09-20T19:48:13.776963906Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=provisioning.alerting t=2024-09-20T19:48:13.798787022Z level=info msg="starting to provision alerting" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=provisioning.alerting t=2024-09-20T19:48:13.798809535Z level=info msg="finished to provision alerting" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=ngalert.state.manager t=2024-09-20T19:48:13.798925533Z level=info msg="Warming state cache for startup" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=grafanaStorageLogger t=2024-09-20T19:48:13.800819205Z level=info msg="storage starting" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=http.server t=2024-09-20T19:48:13.80567753Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=ngalert.state.manager t=2024-09-20T19:48:13.809853752Z level=info msg="State cache has been initialized" states=0 duration=10.926801ms 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=ticker t=2024-09-20T19:48:13.809924136Z level=info msg=starting first_tick=2024-09-20T19:48:20Z 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=ngalert.multiorg.alertmanager t=2024-09-20T19:48:13.809913874Z level=info msg="starting MultiOrg Alertmanager" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore.transactions t=2024-09-20T19:48:13.846502911Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T19:48:13.888 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore.transactions t=2024-09-20T19:48:13.886625662Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T19:48:14.510 INFO:teuthology.orchestra.run.smithi184.stdout:133143986203 2024-09-20T19:48:14.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T19:48:15.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:15 smithi196 ceph-mon[25662]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:15.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:15 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:15.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:15 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[23692]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[30019]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:15 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:16.647 INFO:teuthology.orchestra.run.smithi184.stdout:201863462930 2024-09-20T19:48:16.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T19:48:17.068 INFO:teuthology.orchestra.run.smithi184.stdout:111669149726 2024-09-20T19:48:17.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T19:48:17.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:17 smithi184 ceph-mon[23692]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:17.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:17 smithi184 ceph-mon[30019]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:17 smithi196 ceph-mon[25662]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:17.967 INFO:teuthology.orchestra.run.smithi184.stdout:236223201293 2024-09-20T19:48:17.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T19:48:18.511 INFO:teuthology.orchestra.run.smithi184.stdout:55834574889 2024-09-20T19:48:18.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T19:48:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:19 smithi184 ceph-mon[23692]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:19 smithi184 ceph-mon[30019]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:19.863 INFO:teuthology.orchestra.run.smithi184.stdout:34359738416 2024-09-20T19:48:19.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T19:48:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:19 smithi196 ceph-mon[25662]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:20.228 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:20.228 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:20.739 INFO:teuthology.orchestra.run.smithi184.stdout:77309411365 2024-09-20T19:48:20.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T19:48:20.759 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:21.687 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:21 smithi184 ceph-mon[23692]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:21.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:21 smithi184 ceph-mon[30019]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:21 smithi196 ceph-mon[25662]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:22.199 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:22.199 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:22.200 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:22.219 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:22.219 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:22] "GET /metrics HTTP/1.1" 200 26202 "" "Prometheus/2.43.0" 2024-09-20T19:48:22.298 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:22 smithi184 bash[52625]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T19:48:23.601 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:23 smithi184 ceph-mon[23692]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:23.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:23 smithi184 ceph-mon[30019]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:23 smithi196 ceph-mon[25662]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:24.152 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:23 smithi184 bash[52625]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T19:48:24.497 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:24 smithi184 bash[52625]: Writing manifest to image destination 2024-09-20T19:48:24.798 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:24 smithi184 podman[52625]: 2024-09-20 19:48:24.702105989 +0000 UTC m=+3.735314631 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T19:48:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:25 smithi184 ceph-mon[23692]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:25 smithi184 ceph-mon[30019]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:25 smithi196 ceph-mon[25662]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:26.967 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:26 smithi184 podman[52625]: 2024-09-20 19:48:26.861989516 +0000 UTC m=+5.895198145 container create cfe7897b375d83d2a005fd33deda2cc8ea242bf975cc46ead884e45bc917803e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 podman[52625]: 2024-09-20 19:48:27.203444116 +0000 UTC m=+6.236652758 container init cfe7897b375d83d2a005fd33deda2cc8ea242bf975cc46ead884e45bc917803e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 podman[52625]: 2024-09-20 19:48:27.207237045 +0000 UTC m=+6.240445678 container start cfe7897b375d83d2a005fd33deda2cc8ea242bf975cc46ead884e45bc917803e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.234Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.234Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.235Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.235Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.235Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.235Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T19:48:27.299 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T19:48:27.300 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T19:48:27.301 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T19:48:27.302 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T19:48:27.303 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-a[53104]: ts=2024-09-20T19:48:27.236Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T19:48:27.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-mon[23692]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:27.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:27 smithi184 ceph-mon[30019]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:27.703 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 bash[52625]: cfe7897b375d83d2a005fd33deda2cc8ea242bf975cc46ead884e45bc917803e 2024-09-20T19:48:27.703 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Sep 20 19:48:27 smithi184 systemd[1]: Started Ceph node-exporter.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:48:27.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:27 smithi196 ceph-mon[25662]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:28.436 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 systemd[1]: Starting Ceph node-exporter.b for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[23692]: Deploying daemon node-exporter.b on smithi196 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:28 smithi184 ceph-mon[30019]: Deploying daemon node-exporter.b on smithi196 2024-09-20T19:48:28.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:28 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:28 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:28 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:28.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:28 smithi196 ceph-mon[25662]: Deploying daemon node-exporter.b on smithi196 2024-09-20T19:48:28.768 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 bash[47615]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T19:48:28.768 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 bash[47615]: Getting image source signatures 2024-09-20T19:48:29.140 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 bash[47615]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T19:48:29.140 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 bash[47615]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T19:48:29.140 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:28 smithi196 bash[47615]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T19:48:29.283 INFO:teuthology.orchestra.run.smithi184.stdout:55834574892 2024-09-20T19:48:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[23692]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2368388792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T19:48:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1847158099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T19:48:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[30019]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2368388792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T19:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1847158099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T19:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:29 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:29.842 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574892 for osd.1 2024-09-20T19:48:29.842 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:29 smithi196 ceph-mon[25662]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:29 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2368388792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T19:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:29 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1847158099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T19:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:29 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:30.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 bash[47615]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T19:48:30.390 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 bash[47615]: Writing manifest to image destination 2024-09-20T19:48:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:167503724570 2024-09-20T19:48:30.848 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 podman[47615]: 2024-09-20 19:48:30.471390523 +0000 UTC m=+2.127498327 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 podman[47615]: 2024-09-20 19:48:30.569847852 +0000 UTC m=+2.225955655 container create edb7c108b4f8ac9f76163f23a3f5a76231a92b419adb40c4054f004678d55e7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 podman[47615]: 2024-09-20 19:48:30.755585616 +0000 UTC m=+2.411693427 container init edb7c108b4f8ac9f76163f23a3f5a76231a92b419adb40c4054f004678d55e7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 podman[47615]: 2024-09-20 19:48:30.75871605 +0000 UTC m=+2.414823856 container start edb7c108b4f8ac9f76163f23a3f5a76231a92b419adb40c4054f004678d55e7f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T19:48:30.849 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.762Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T19:48:30.850 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T19:48:30.851 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T19:48:30.852 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T19:48:30.853 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T19:48:30.854 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T19:48:30.854 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.763Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T19:48:30.854 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.764Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T19:48:30.854 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-node-exporter-b[47691]: ts=2024-09-20T19:48:30.764Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T19:48:30.854 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 bash[47615]: edb7c108b4f8ac9f76163f23a3f5a76231a92b419adb40c4054f004678d55e7f 2024-09-20T19:48:31.139 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Sep 20 19:48:30 smithi196 systemd[1]: Started Ceph node-exporter.b for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:48:31.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724570 for osd.5 2024-09-20T19:48:31.362 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:48:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:48:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:48:32.203 INFO:teuthology.orchestra.run.smithi184.stdout:236223201298 2024-09-20T19:48:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:32] "GET /metrics HTTP/1.1" 200 26211 "" "Prometheus/2.43.0" 2024-09-20T19:48:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:32 smithi184 ceph-mon[23692]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:48:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:32 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3179783643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T19:48:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:32 smithi184 ceph-mon[30019]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:32 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3179783643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T19:48:32.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:32 smithi196 ceph-mon[25662]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:48:32.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:32 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3179783643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T19:48:33.545 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201293 got 236223201298 for osd.7 2024-09-20T19:48:33.546 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:33.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:33 smithi196 ceph-mon[25662]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:33.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:33 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:33 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:33 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[23692]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[30019]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:33 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:34.263 INFO:teuthology.orchestra.run.smithi184.stdout:77309411368 2024-09-20T19:48:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:34 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/202830910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T19:48:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:34 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:34 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/202830910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T19:48:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:34 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:34 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/202830910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T19:48:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:34 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:34.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-09-20T19:48:34.959 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:35 smithi184 ceph-mon[23692]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:35 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:35 smithi184 ceph-mon[30019]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:35 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:35.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:35 smithi196 ceph-mon[25662]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:35.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:35 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:37.620 INFO:teuthology.orchestra.run.smithi184.stdout:34359738420 2024-09-20T19:48:37.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:37 smithi184 ceph-mon[23692]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:37.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:37 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:37.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:37 smithi184 ceph-mon[30019]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:37.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:37 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:37 smithi196 ceph-mon[25662]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:37 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:38.166 INFO:teuthology.orchestra.run.smithi184.stdout:133143986208 2024-09-20T19:48:38.384 INFO:teuthology.orchestra.run.smithi184.stdout:111669149731 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4094273309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1192123582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3337988747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/4094273309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T19:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1192123582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T19:48:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:38 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3337988747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T19:48:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:38 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/4094273309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T19:48:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:38 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:38 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1192123582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T19:48:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:38 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3337988747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T19:48:39.168 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738420 for osd.0 2024-09-20T19:48:39.168 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:39.736 INFO:teuthology.orchestra.run.smithi184.stdout:201863462935 2024-09-20T19:48:39.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:39 smithi184 ceph-mon[23692]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:39.737 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:39 smithi184 ceph-mon[30019]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:39 smithi196 ceph-mon[25662]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:40.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986208 for osd.4 2024-09-20T19:48:40.134 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:40.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:40 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/460115623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T19:48:40.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:40 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/460115623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T19:48:40.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149726 got 111669149731 for osd.3 2024-09-20T19:48:40.778 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:40 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/460115623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T19:48:41.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462935 for osd.6 2024-09-20T19:48:41.196 DEBUG:teuthology.parallel:result is None 2024-09-20T19:48:41.196 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T19:48:41.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:48:41.822 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:41 smithi184 ceph-mon[23692]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:41.822 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:41 smithi184 ceph-mon[30019]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:41 smithi196 ceph-mon[25662]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:42.141 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:42] "GET /metrics HTTP/1.1" 200 26211 "" "Prometheus/2.43.0" 2024-09-20T19:48:42.161 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:43 smithi196 ceph-mon[25662]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:43.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:43 smithi184 ceph-mon[23692]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:43.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:43 smithi184 ceph-mon[30019]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:44.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:44.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:44.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[30019]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[23692]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:45.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:45 smithi196 ceph-mon[25662]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:46.362 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:48:46.363 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2024-09-20T19:48:46.972 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-20T19:48:44.401895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226612,"kb_used_data":6364,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734604,"statfs":{"total":767960285184,"available":767728234496,"internally_reserved":0,"allocated":6516736,"data_stored":3701058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":255,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T19:47:46.457325+0000","last_change":"2024-09-20T19:47:17.540109+0000","last_active":"2024-09-20T19:47:46.457325+0000","last_peered":"2024-09-20T19:47:46.457325+0000","last_clean":"2024-09-20T19:47:46.457325+0000","last_became_active":"2024-09-20T19:47:17.539748+0000","last_became_peered":"2024-09-20T19:47:17.539748+0000","last_unstale":"2024-09-20T19:47:46.457325+0000","last_undegraded":"2024-09-20T19:47:46.457325+0000","last_fullsized":"2024-09-20T19:47:46.457325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:45:22.675858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:45:22.675858+0000","last_clean_scrub_stamp":"2024-09-20T19:45:22.675858+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T00:52:14.469483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716972,"statfs":{"total":95995035648,"available":95966179328,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48899999999999999}]},{"osd":4,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.501},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53600000000000003}]},{"osd":5,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.47799999999999998}]},{"osd":6,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.51200000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":831152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.63400000000000001}]},{"osd":1,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.495}]},{"osd":4,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.312}]},{"osd":5,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34200000000000003}]},{"osd":7,"last update":"Fri Sep 20 19:48:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.41399999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60799999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.379},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.496},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":0.69499999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68999999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65000000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36799999999999999}]},{"osd":2,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38300000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47999999999999998}]},{"osd":3,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.66800000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.378},"min":{"1min":0.32000000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.28699999999999998}]},{"osd":6,"last update":"Fri Sep 20 19:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42199999999999999}]},{"osd":7,"last update":"Fri Sep 20 19:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.495},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.45400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66500000000000004,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.437}]},{"osd":2,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61799999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89100000000000001,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.42699999999999999}]},{"osd":5,"last update":"Fri Sep 20 19:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67300000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.35199999999999998}]},{"osd":6,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40500000000000003}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.46899999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.434,"15min":0.443},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53000000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42499999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.57299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.35599999999999998}]},{"osd":2,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.46500000000000002,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61499999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.379},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.42999999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.34599999999999997}]},{"osd":4,"last update":"Fri Sep 20 19:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.75,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Fri Sep 20 19:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72799999999999998,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46100000000000002}]},{"osd":6,"last update":"Fri Sep 20 19:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Sep 20 19:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.504},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.46600000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.5,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34899999999999998},"min":{"1min":0.28899999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.48899999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.28599999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.502,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.313}]},{"osd":3,"last update":"Fri Sep 20 19:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.46000000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.372}]},{"osd":4,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71099999999999997,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.73699999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Sep 20 19:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.66200000000000003,"5min":0.749,"15min":0.749},"last":0.44},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72799999999999998,"5min":0.753,"15min":0.753},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Sep 20 19:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":589824,"data_stored":241115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.23000000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.53700000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.37},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64000000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439}]},{"osd":3,"last update":"Fri Sep 20 19:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.503},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55200000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54000000000000004}]},{"osd":4,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.55900000000000005},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.627}]},{"osd":5,"last update":"Fri Sep 20 19:48:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.58399999999999996,"15min":0.56899999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.8420000000000001,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Sep 20 19:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58999999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1183744,"data_stored":831483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 19:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.20799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53100000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.32000000000000001},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54900000000000004,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Sep 20 19:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56999999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.61399999999999999}]},{"osd":4,"last update":"Fri Sep 20 19:48:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39200000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.36699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80000000000000004,"5min":0.873,"15min":0.873},"last":0.51100000000000001}]},{"osd":5,"last update":"Fri Sep 20 19:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.55000000000000004}]},{"osd":6,"last update":"Fri Sep 20 19:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.59699999999999998}]},{"osd":7,"last update":"Fri Sep 20 19:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:48:46.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:48:47.219 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:47.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:47 smithi184 ceph-mon[23692]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:47.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:47 smithi184 ceph-mon[23692]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:47.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:47 smithi184 ceph-mon[30019]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:47.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:47 smithi184 ceph-mon[30019]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:47 smithi196 ceph-mon[25662]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:47 smithi196 ceph-mon[25662]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:49.884 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:49 smithi184 ceph-mon[23692]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:49.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:49 smithi184 ceph-mon[30019]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:49 smithi196 ceph-mon[25662]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:50.268 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:48:50.268 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2024-09-20T19:48:51.120 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-20T19:48:48.402543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226612,"kb_used_data":6364,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734604,"statfs":{"total":767960285184,"available":767728234496,"internally_reserved":0,"allocated":6516736,"data_stored":3701058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001694"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":255,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T19:47:46.457325+0000","last_change":"2024-09-20T19:47:17.540109+0000","last_active":"2024-09-20T19:47:46.457325+0000","last_peered":"2024-09-20T19:47:46.457325+0000","last_clean":"2024-09-20T19:47:46.457325+0000","last_became_active":"2024-09-20T19:47:17.539748+0000","last_became_peered":"2024-09-20T19:47:17.539748+0000","last_unstale":"2024-09-20T19:47:46.457325+0000","last_undegraded":"2024-09-20T19:47:46.457325+0000","last_fullsized":"2024-09-20T19:47:46.457325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:45:22.675858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:45:22.675858+0000","last_clean_scrub_stamp":"2024-09-20T19:45:22.675858+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T00:52:14.469483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716972,"statfs":{"total":95995035648,"available":95966179328,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.501},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.56599999999999995}]},{"osd":1,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.62}]},{"osd":3,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.60199999999999998}]},{"osd":4,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.47499999999999998}]},{"osd":5,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.58699999999999997}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":831152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.497}]},{"osd":2,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Fri Sep 20 19:48:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.56399999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60799999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.496},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":0.69499999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68999999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.375},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65000000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38300000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.65300000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.66800000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Fri Sep 20 19:48:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.378},"min":{"1min":0.32000000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.437},{"interface":"front","average":{"1min":0.439,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.42099999999999999}]},{"osd":6,"last update":"Fri Sep 20 19:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.32400000000000001}]},{"osd":7,"last update":"Fri Sep 20 19:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.35899999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66500000000000004,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61799999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58299999999999996,"15min":0.59399999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.89100000000000001,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Sep 20 19:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67300000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Sep 20 19:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.443},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.65500000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.245},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.434,"15min":0.443},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53000000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42499999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.57299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.313}]},{"osd":2,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.46500000000000002,"15min":0.47099999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61499999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42799999999999999,"15min":0.42999999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Sep 20 19:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.75,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Sep 20 19:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57599999999999996,"15min":0.57999999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72799999999999998,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47499999999999998}]},{"osd":6,"last update":"Fri Sep 20 19:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53900000000000003}]},{"osd":7,"last update":"Fri Sep 20 19:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.51300000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.5,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34899999999999998},"min":{"1min":0.28899999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.48899999999999999,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.28599999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.502,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.46700000000000003}]},{"osd":3,"last update":"Fri Sep 20 19:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.46000000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.504}]},{"osd":4,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71099999999999997,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.73699999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.627}]},{"osd":5,"last update":"Fri Sep 20 19:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.66200000000000003,"5min":0.749,"15min":0.749},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72799999999999998,"5min":0.753,"15min":0.753},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Sep 20 19:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":589824,"data_stored":241115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 19:47:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.23000000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.52200000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.53700000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.311}]},{"osd":2,"last update":"Fri Sep 20 19:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.37},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64000000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.33900000000000002}]},{"osd":3,"last update":"Fri Sep 20 19:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55200000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28199999999999997}]},{"osd":4,"last update":"Fri Sep 20 19:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.57499999999999996,"15min":0.55900000000000005},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.376}]},{"osd":5,"last update":"Fri Sep 20 19:48:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.58399999999999996,"15min":0.56899999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.8420000000000001,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.38600000000000001}]},{"osd":6,"last update":"Fri Sep 20 19:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.443}]},{"osd":7,"last update":"Fri Sep 20 19:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.41899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1183744,"data_stored":831483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 19:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.20799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53100000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.32000000000000001},"min":{"1min":0.26100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54900000000000004,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Fri Sep 20 19:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.252},{"interface":"front","average":{"1min":0.373,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56999999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33600000000000002}]},{"osd":3,"last update":"Fri Sep 20 19:47:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Sep 20 19:48:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39200000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.36699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80000000000000004,"5min":0.873,"15min":0.873},"last":0.65300000000000002}]},{"osd":5,"last update":"Fri Sep 20 19:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.66400000000000003}]},{"osd":6,"last update":"Fri Sep 20 19:47:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Sep 20 19:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.746},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:48:51.121 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T19:48:51.121 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T19:48:51.121 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T19:48:51.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:48:51.551 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:51.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:51 smithi184 ceph-mon[23692]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:51.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:51 smithi184 ceph-mon[23692]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:51.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:51 smithi184 ceph-mon[30019]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:51.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:51 smithi184 ceph-mon[30019]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:51 smithi196 ceph-mon[25662]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:48:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:51 smithi196 ceph-mon[25662]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:48:52] "GET /metrics HTTP/1.1" 200 26209 "" "Prometheus/2.43.0" 2024-09-20T19:48:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[30019]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[23692]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:53 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:53 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:53 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:53 smithi196 ceph-mon[25662]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:53 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:53 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:53.971 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-20T19:48:53.971 INFO:teuthology.orchestra.run.smithi184.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T19:48:54.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:54 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3418779436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:48:54.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:54 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3418779436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:48:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:54 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3418779436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:48:54.642 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T19:48:54.642 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T19:48:54.643 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:48:54.652 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi184.front.sepia.ceph.com 2024-09-20T19:48:54.652 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-20T19:48:55.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:55 smithi196 ceph-mon[25662]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:55 smithi184 ceph-mon[23692]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:55 smithi184 ceph-mon[30019]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:56.059 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:48:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:57 smithi184 ceph-mon[23692]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:57 smithi184 ceph-mon[30019]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:57.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:57 smithi196 ceph-mon[25662]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:48:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[23692]: Reconfiguring daemon alertmanager.a on smithi184 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:48:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:58 smithi184 ceph-mon[30019]: Reconfiguring daemon alertmanager.a on smithi184 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T19:48:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:58 smithi196 ceph-mon[25662]: Reconfiguring daemon alertmanager.a on smithi184 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[23692]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[30019]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:48:59.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:48:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:59 smithi196 ceph-mon[25662]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:49:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:48:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:00 smithi184 ceph-mon[23692]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:00 smithi184 ceph-mon[30019]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:00 smithi196 ceph-mon[25662]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:01.954 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:01 smithi184 systemd[1]: Stopping Ceph alertmanager.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:49:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:49:02] "GET /metrics HTTP/1.1" 200 26285 "" "Prometheus/2.43.0" 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "0476ac8d-2dad-469b-93f8-f8b54500b9a5", 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "r", 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "07647cf7-a1bc-4dd2-a707-3c2414e59807", 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 1 2024-09-20T19:49:02.708 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:49:02.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[50559]: ts=2024-09-20T19:49:02.719Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[23692]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[23692]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T19:49:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T19:49:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[30019]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[30019]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T19:49:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T19:49:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:02 smithi196 ceph-mon[25662]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:03.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:02 smithi196 ceph-mon[25662]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T19:49:03.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T19:49:03.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-20T19:49:03.685 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:03 smithi184 podman[58817]: 2024-09-20 19:49:03.389770745 +0000 UTC m=+0.782589221 container died 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:03.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[23692]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:03.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:49:03.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[23692]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T19:49:03.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[30019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:03.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:49:03.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:03 smithi184 ceph-mon[30019]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T19:49:03.959 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:03 smithi184 podman[58817]: 2024-09-20 19:49:03.731081967 +0000 UTC m=+1.123900433 container cleanup 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:03.959 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:03 smithi184 bash[58817]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T19:49:03.959 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:03 smithi184 podman[58841]: 2024-09-20 19:49:03.882196066 +0000 UTC m=+1.113666735 container remove 02702e873fd7bca86627e5f21ad4ee863744b959bc801d84f96651572c62f601 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:03.959 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:03 smithi184 podman[58841]: 2024-09-20 19:49:03.957851716 +0000 UTC m=+1.189322340 volume remove b3dc3145bec6fa0fce441d09168f13eb6dc6b259c080ac08d929e169bda40af3 2024-09-20T19:49:04.116 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:03 smithi196 ceph-mon[25662]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:03 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1972608488' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T19:49:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:03 smithi196 ceph-mon[25662]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[23692]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[23692]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[23692]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[30019]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[30019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:04 smithi184 ceph-mon[30019]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T19:49:04.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:04 smithi184 podman[58921]: 2024-09-20 19:49:04.535951684 +0000 UTC m=+0.444023613 container died 01f02669dd893c8904c9109f204ec8a330c328c472027a8fef5d491d977e2631 (image=quay.ceph.io/ceph-ci/ceph:reef, name=intelligent_engelbart, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS) 2024-09-20T19:49:04.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:04 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T19:49:04.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:04 smithi184 systemd[1]: Stopped Ceph alertmanager.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:49:04.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:04 smithi184 systemd[1]: Starting Ceph alertmanager.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:49:05.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:04 smithi196 ceph-mon[25662]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:05.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:04 smithi196 ceph-mon[25662]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:05.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:04 smithi196 ceph-mon[25662]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T19:49:05.854 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:05 smithi184 podman[59048]: 2024-09-20 19:49:05.591370727 +0000 UTC m=+0.247268139 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T19:49:05.854 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:05 smithi184 podman[59048]: 2024-09-20 19:49:05.692288292 +0000 UTC m=+0.348185733 volume create 5625a4169f168e491092566785c31d4cbfd520c39884fe3ce4f2b0fc39452db9 2024-09-20T19:49:05.903 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:49:05.903 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "8c038f97-9a0b-43ec-9870-1c40ad5f12d9", 2024-09-20T19:49:05.903 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2024-09-20T19:49:05.903 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "default", 2024-09-20T19:49:05.903 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": true, 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "", 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [] 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "0476ac8d-2dad-469b-93f8-f8b54500b9a5", 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2024-09-20T19:49:05.904 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2024-09-20T19:49:05.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:05.905 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled_features": [ 2024-09-20T19:49:05.905 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2024-09-20T19:49:05.905 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:49:05.905 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:49:06.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:05 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:05 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:05 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:05 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:06.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-20T19:49:06.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 podman[59048]: 2024-09-20 19:49:06.633754311 +0000 UTC m=+1.289651712 container create 3d35fb41de6ec76dd2111c949b62d19fdceadbc0957a6c5c1d86e79818d59576 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:07.021 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:07.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:07 smithi184 ceph-mon[23692]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:07.143 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:07 smithi184 ceph-mon[30019]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 podman[59048]: 2024-09-20 19:49:06.886136951 +0000 UTC m=+1.542034354 container init 3d35fb41de6ec76dd2111c949b62d19fdceadbc0957a6c5c1d86e79818d59576 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 podman[59048]: 2024-09-20 19:49:06.889114075 +0000 UTC m=+1.545011474 container start 3d35fb41de6ec76dd2111c949b62d19fdceadbc0957a6c5c1d86e79818d59576 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.901Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.901Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.902Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.184 port=9094 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.903Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.926Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.927Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.929Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T19:49:07.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:06.929Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T19:49:07.144 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 bash[59048]: 3d35fb41de6ec76dd2111c949b62d19fdceadbc0957a6c5c1d86e79818d59576 2024-09-20T19:49:07.144 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:06 smithi184 systemd[1]: Started Ceph alertmanager.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:49:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:07 smithi196 ceph-mon[25662]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:08.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[23692]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:49:08.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[23692]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:49:08.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[30019]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:49:08.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-mon[30019]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:08 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:08 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:08 smithi196 ceph-mon[25662]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:08 smithi196 ceph-mon[25662]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:49:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:09 smithi184 ceph-mon[23692]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:09 smithi184 ceph-mon[30019]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:09.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:08.903Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000115321s 2024-09-20T19:49:09.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:09 smithi196 ceph-mon[25662]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "276c84a4-5a3d-46c3-a38c-bea1290656a4", 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "z", 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "z.rgw.meta:root", 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "z.rgw.control", 2024-09-20T19:49:09.841 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "z.rgw.log", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "z.rgw.otp", 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2024-09-20T19:49:09.842 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-20T19:49:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "0476ac8d-2dad-469b-93f8-f8b54500b9a5", 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-20T19:49:09.844 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:49:10.502 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 systemd[1]: Stopping Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:49:10.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-20T19:49:10.738 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:10.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:49:10.820 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.501Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.502Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.502Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[44781]: ts=2024-09-20T19:49:10.502Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T19:49:10.821 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:10 smithi196 podman[49094]: 2024-09-20 19:49:10.511792712 +0000 UTC m=+0.144471215 container died f4b01811eefce5f7419ab947bf53e62588ddba77956815377fc79ce754bee253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:11 smithi184 ceph-mon[23692]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 2 op/s 2024-09-20T19:49:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:11 smithi184 ceph-mon[30019]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 2 op/s 2024-09-20T19:49:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-mon[25662]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 2 op/s 2024-09-20T19:49:11.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49094]: 2024-09-20 19:49:11.010328292 +0000 UTC m=+0.643006849 container cleanup f4b01811eefce5f7419ab947bf53e62588ddba77956815377fc79ce754bee253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:11.140 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 bash[49094]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:49:11.639 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49105]: 2024-09-20 19:49:11.361908006 +0000 UTC m=+0.852083460 container remove f4b01811eefce5f7419ab947bf53e62588ddba77956815377fc79ce754bee253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:11.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T19:49:11.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 systemd[1]: Stopped Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:49:11.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.197s CPU time. 2024-09-20T19:49:11.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 systemd[1]: Starting Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:49:11.949 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49200]: 2024-09-20 19:49:11.599742464 +0000 UTC m=+0.017084722 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T19:49:11.949 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49200]: 2024-09-20 19:49:11.729533063 +0000 UTC m=+0.146875313 container create 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Bus STOPPING 2024-09-20T19:49:12.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49200]: 2024-09-20 19:49:11.948328077 +0000 UTC m=+0.365670332 container init 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:12.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 podman[49200]: 2024-09-20 19:49:11.951181465 +0000 UTC m=+0.368523720 container start 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:49:12.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.975Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T19:49:12.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.975Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T19:49:12.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.975Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.975Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.975Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.977Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.977Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.978Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.978Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.980Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.980Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.614µs 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.980Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.981Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.982Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.982Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.237µs wal_replay_duration=1.35846ms wbl_replay_duration=102ns total_replay_duration=1.400818ms 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.982Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:49:12.391 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.982Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:11.982Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:12.003Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.139104ms db_storage=752ns remote_storage=1.118µs web_handler=332ns query_engine=539ns scrape=383.522µs scrape_sd=90.586µs notify=16.971µs notify_sd=18.733µs rules=19.326681ms tracing=6.757µs 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:12.003Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:49:12.003Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:12 smithi196 bash[49200]: 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb 2024-09-20T19:49:12.392 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:49:12 smithi196 systemd[1]: Started Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:49:12.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:49:12.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Bus STOPPED 2024-09-20T19:49:12.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Bus STARTING 2024-09-20T19:49:12.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Serving on http://:::9283 2024-09-20T19:49:12.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Bus STARTED 2024-09-20T19:49:12.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:12] ENGINE Bus STOPPING 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:13.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-20T19:49:13.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:13 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:49:13.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STOPPED 2024-09-20T19:49:13.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STARTING 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Serving on http://:::9283 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STARTED 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STOPPING 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:13.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.9 KiB/s wr, 4 op/s 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi184.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: pgmap v51: 33 pgs: 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-20T19:49:13.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:14.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:49:14.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STOPPED 2024-09-20T19:49:14.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STARTING 2024-09-20T19:49:14.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Serving on http://:::9283 2024-09-20T19:49:14.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:13] ENGINE Bus STARTED 2024-09-20T19:49:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:14 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:49:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:14 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:49:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:14 smithi196 ceph-mon[25662]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:49:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T19:49:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[23692]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[30019]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T19:49:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[23692]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:15 smithi184 ceph-mon[30019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:15 smithi196 ceph-mon[25662]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:16 smithi196 ceph-mon[25662]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:49:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:16 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T19:49:16.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:16 smithi184 ceph-mon[23692]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:49:16.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:16 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T19:49:16.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:16 smithi184 ceph-mon[30019]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T19:49:16.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:16 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T19:49:17.174 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:49:16 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:49:16.905Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002403115s 2024-09-20T19:49:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:17 smithi196 ceph-mon[25662]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-09-20T19:49:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T19:49:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:17 smithi196 ceph-mon[25662]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:49:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[30019]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-09-20T19:49:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T19:49:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[30019]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:49:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[23692]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 4 op/s 2024-09-20T19:49:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1488136116' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T19:49:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:17 smithi184 ceph-mon[23692]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T19:49:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[30019]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:49:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[23692]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:49:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:18 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:18.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:18 smithi196 ceph-mon[25662]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T19:49:18.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:18 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:18.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:18 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "622517ca-b9bc-4609-b54c-84f20b121ba4", 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 1, 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "07647cf7-a1bc-4dd2-a707-3c2414e59807", 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2024-09-20T19:49:19.251 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "622517ca-b9bc-4609-b54c-84f20b121ba4", 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "8c038f97-9a0b-43ec-9870-1c40ad5f12d9", 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "default", 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": true, 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "276c84a4-5a3d-46c3-a38c-bea1290656a4", 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:19.252 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "276c84a4-5a3d-46c3-a38c-bea1290656a4", 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "z", 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": false, 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:49:19.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "0476ac8d-2dad-469b-93f8-f8b54500b9a5", 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2024-09-20T19:49:19.254 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled_features": [ 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "276c84a4-5a3d-46c3-a38c-bea1290656a4", 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791112963 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "8c038f97-9a0b-43ec-9870-1c40ad5f12d9", 2024-09-20T19:49:19.255 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "276c84a4-5a3d-46c3-a38c-bea1290656a4", 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2024-09-20T19:49:19.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "user_ratelimit": { 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_ratelimit": { 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.257 INFO:teuthology.orchestra.run.smithi184.stdout: "anonymous_ratelimit": { 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "0476ac8d-2dad-469b-93f8-f8b54500b9a5", 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "r", 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2024-09-20T19:49:19.258 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:49:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:19 smithi196 ceph-mon[25662]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s 2024-09-20T19:49:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:19 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:49:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:19 smithi196 ceph-mon[25662]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:49:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:19 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:19 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[23692]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s 2024-09-20T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[23692]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[30019]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s 2024-09-20T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[30019]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3909315482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:19 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T19:49:19.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-20T19:49:20.094 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:20.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:20 smithi184 ceph-mon[23692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:49:20.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:20 smithi184 ceph-mon[23692]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:49:20.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:20 smithi184 ceph-mon[30019]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:49:20.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:20 smithi184 ceph-mon[30019]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:49:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:20 smithi196 ceph-mon[25662]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T19:49:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:20 smithi196 ceph-mon[25662]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T19:49:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[23692]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[23692]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:49:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[30019]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[30019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:21 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:49:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:21 smithi196 ceph-mon[25662]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:21 smithi196 ceph-mon[25662]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:21 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:49:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:49:22] "GET /metrics HTTP/1.1" 200 30893 "" "Prometheus/2.43.0" 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:22 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.579 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled rgw.foo update... 2024-09-20T19:49:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:22 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:22 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:22 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-20T19:49:23.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[23692]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:49:23.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:23 smithi184 ceph-mon[30019]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:49:23.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-20T19:49:23.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:23.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:23 smithi196 ceph-mon[25662]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:49:24.206 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply rgw smpl' 2024-09-20T19:49:25.123 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:25 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:25.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T19:49:25.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:25.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:25 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:26 smithi196 ceph-mon[25662]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:49:26.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:26 smithi184 ceph-mon[23692]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:49:26.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:26 smithi184 ceph-mon[30019]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:49:27.290 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled rgw.smpl update... 2024-09-20T19:49:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:27 smithi184 ceph-mon[23692]: pgmap v65: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 144 op/s 2024-09-20T19:49:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:27 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:27 smithi184 ceph-mon[30019]: pgmap v65: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 144 op/s 2024-09-20T19:49:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:27 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:27 smithi196 ceph-mon[25662]: pgmap v65: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 5.6 KiB/s wr, 144 op/s 2024-09-20T19:49:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:27 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:28.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph osd pool create foo' 2024-09-20T19:49:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:28 smithi196 ceph-mon[25662]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:28 smithi196 ceph-mon[25662]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:28.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:28 smithi184 ceph-mon[23692]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:28.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:28 smithi184 ceph-mon[23692]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:28.853 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:28 smithi184 ceph-mon[30019]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:28.854 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:28 smithi184 ceph-mon[30019]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:28.919 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:29 smithi184 ceph-mon[23692]: pgmap v66: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.5 KiB/s wr, 115 op/s 2024-09-20T19:49:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:29 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:29 smithi184 ceph-mon[30019]: pgmap v66: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.5 KiB/s wr, 115 op/s 2024-09-20T19:49:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:29 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:29 smithi196 ceph-mon[25662]: pgmap v66: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.5 KiB/s wr, 115 op/s 2024-09-20T19:49:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:29 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:31.606 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'foo' created 2024-09-20T19:49:31.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: pgmap v67: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.1 KiB/s wr, 105 op/s 2024-09-20T19:49:31.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T19:49:31.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: pgmap v67: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.1 KiB/s wr, 105 op/s 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T19:49:31.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:31.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: pgmap v67: 129 pgs: 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.1 KiB/s wr, 105 op/s 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:31 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:31.916 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:31] ENGINE Bus STOPPING 2024-09-20T19:49:32.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:49:32] "GET /metrics HTTP/1.1" 200 35643 "" "Prometheus/2.43.0" 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STOPPED 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STARTING 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Serving on http://:::9283 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STARTED 2024-09-20T19:49:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STOPPING 2024-09-20T19:49:32.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:32.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:32 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STOPPED 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STARTING 2024-09-20T19:49:32.866 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Serving on http://:::9283 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:49:32] ENGINE Bus STARTED 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: Saving service rgw.foo spec with placement count:2 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3204584261' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:32.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:32 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:32.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'rbd pool init foo' 2024-09-20T19:49:33.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:33.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:33.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:33.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.3 KiB/s wr, 111 op/s 2024-09-20T19:49:33.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:49:33.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:33 smithi196 ceph-mon[25662]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.3 KiB/s wr, 111 op/s 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[23692]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: Checking dashboard <-> RGW credentials 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.3 KiB/s wr, 111 op/s 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T19:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:33 smithi184 ceph-mon[30019]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T19:49:34.657 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:34 smithi196 ceph-mon[25662]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:49:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:34 smithi196 ceph-mon[25662]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:34 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[23692]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[23692]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[30019]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[30019]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:34.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:34 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:36 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:36 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:36 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:37 smithi184 ceph-mon[23692]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:37 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T19:49:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:37 smithi184 ceph-mon[30019]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:37 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T19:49:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:37 smithi196 ceph-mon[25662]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:37 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T19:49:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:38 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T19:49:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:38 smithi184 ceph-mon[23692]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:49:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:38 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T19:49:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:38 smithi184 ceph-mon[30019]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:49:38.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:38 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/790694869' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T19:49:38.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:38 smithi196 ceph-mon[25662]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T19:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[23692]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[23692]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 553 B/s wr, 187 op/s 2024-09-20T19:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[30019]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[30019]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 553 B/s wr, 187 op/s 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:39 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:39 smithi196 ceph-mon[25662]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T19:49:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:39 smithi196 ceph-mon[25662]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 553 B/s wr, 187 op/s 2024-09-20T19:49:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:39 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:39 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:39 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:39.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-20T19:49:40.080 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:40.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[23692]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:40.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[23692]: Cluster is now healthy 2024-09-20T19:49:40.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[23692]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:49:40.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[30019]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:40.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[30019]: Cluster is now healthy 2024-09-20T19:49:40.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:40 smithi184 ceph-mon[30019]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:49:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:40 smithi196 ceph-mon[25662]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T19:49:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:40 smithi196 ceph-mon[25662]: Cluster is now healthy 2024-09-20T19:49:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:40 smithi196 ceph-mon[25662]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T19:49:41.250 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:41 smithi184 ceph-mon[23692]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:41.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:41 smithi184 ceph-mon[30019]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:41 smithi196 ceph-mon[25662]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 511 B/s wr, 173 op/s 2024-09-20T19:49:41.640 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled iscsi.foo update... 2024-09-20T19:49:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:49:42] "GET /metrics HTTP/1.1" 200 35643 "" "Prometheus/2.43.0" 2024-09-20T19:49:42.339 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'sleep 120' 2024-09-20T19:49:42.817 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[23692]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[23692]: Saving service iscsi.foo spec with placement count:1 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[23692]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[30019]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[30019]: Saving service iscsi.foo spec with placement count:1 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:42 smithi184 ceph-mon[30019]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:42 smithi196 ceph-mon[25662]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:49:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:42 smithi196 ceph-mon[25662]: Saving service iscsi.foo spec with placement count:1 2024-09-20T19:49:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:42 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:42 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:49:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:42 smithi196 ceph-mon[25662]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:49:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-20T19:49:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 334 B/s wr, 0 op/s 2024-09-20T19:49:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T19:49:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T19:49:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[23692]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 334 B/s wr, 0 op/s 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-20T19:49:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:45 smithi184 ceph-mon[30019]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 334 B/s wr, 0 op/s 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-20T19:49:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:45 smithi196 ceph-mon[25662]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T19:49:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:46 smithi196 ceph-mon[25662]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:49:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[23692]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:49:47.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[30019]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T19:49:47.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[23692]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[30019]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:47 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:47 smithi196 ceph-mon[25662]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:49:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:47 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:47 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:49.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:49 smithi184 ceph-mon[23692]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:49.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:49 smithi184 ceph-mon[30019]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:49 smithi196 ceph-mon[25662]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:51.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[30019]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:49:51.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:51.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:50 smithi184 ceph-mon[23692]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:50 smithi196 ceph-mon[25662]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:49:51.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:51 smithi184 ceph-mon[23692]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:49:51.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:51 smithi184 ceph-mon[30019]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:51 smithi196 ceph-mon[25662]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:49:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:49:52] "GET /metrics HTTP/1.1" 200 38121 "" "Prometheus/2.43.0" 2024-09-20T19:49:52.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:52 smithi184 ceph-mon[23692]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 138 B/s, 0 objects/s recovering 2024-09-20T19:49:52.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:52 smithi184 ceph-mon[30019]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 138 B/s, 0 objects/s recovering 2024-09-20T19:49:53.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:52 smithi196 ceph-mon[25662]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 138 B/s, 0 objects/s recovering 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[23692]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:54.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:54 smithi184 ceph-mon[30019]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:49:54.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:54.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:54 smithi196 ceph-mon[25662]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:49:55.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:55 smithi196 ceph-mon[25662]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 114 B/s, 0 objects/s recovering 2024-09-20T19:49:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:55 smithi184 ceph-mon[30019]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 114 B/s, 0 objects/s recovering 2024-09-20T19:49:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:55 smithi184 ceph-mon[23692]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 114 B/s, 0 objects/s recovering 2024-09-20T19:49:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 100 B/s wr, 97 op/s; 103 B/s, 0 objects/s recovering 2024-09-20T19:49:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:57 smithi196 ceph-mon[25662]: Deploying daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:49:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 100 B/s wr, 97 op/s; 103 B/s, 0 objects/s recovering 2024-09-20T19:49:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[30019]: Deploying daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 100 B/s wr, 97 op/s; 103 B/s, 0 objects/s recovering 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: Saving service rgw.smpl spec with placement count:2 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:49:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:57 smithi184 ceph-mon[23692]: Deploying daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:49:59.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[23692]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:49:59.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:59.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[30019]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:49:59.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:59.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:49:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:59 smithi196 ceph-mon[25662]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:49:59.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:49:59.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:49:59.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:49:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:00.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[23692]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[23692]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[23692]: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:50:00.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[30019]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[30019]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:00 smithi184 ceph-mon[30019]: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:50:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:00 smithi196 ceph-mon[25662]: Health detail: HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:00 smithi196 ceph-mon[25662]: [WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:50:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:00 smithi196 ceph-mon[25662]: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:01.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 88 B/s wr, 85 op/s; 91 B/s, 0 objects/s recovering 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:02.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:02] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.43.0" 2024-09-20T19:50:02.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[23692]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T19:50:02.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3919884609' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:50:02.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1829282474' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:02.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:02.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[30019]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T19:50:02.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3919884609' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:50:02.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1829282474' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:02.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:02 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:02.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:02 smithi196 ceph-mon[25662]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T19:50:02.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3919884609' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:50:02.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1829282474' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:02.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:02 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]: dispatch 2024-09-20T19:50:03.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 166 op/s; 88 B/s, 0 objects/s recovering 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]': finished 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]: dispatch 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 166 op/s; 88 B/s, 0 objects/s recovering 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]': finished 2024-09-20T19:50:03.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]: dispatch 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:03 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 170 B/s wr, 166 op/s; 88 B/s, 0 objects/s recovering 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/2073022965"}]': finished 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]: dispatch 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:03 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:04 smithi196 ceph-mon[25662]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 204 B/s wr, 199 op/s 2024-09-20T19:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]': finished 2024-09-20T19:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:04 smithi196 ceph-mon[25662]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]: dispatch 2024-09-20T19:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:05.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[23692]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 204 B/s wr, 199 op/s 2024-09-20T19:50:05.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]': finished 2024-09-20T19:50:05.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[23692]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:50:05.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]: dispatch 2024-09-20T19:50:05.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[30019]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 204 B/s wr, 199 op/s 2024-09-20T19:50:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2201331252' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/2073022965"}]': finished 2024-09-20T19:50:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[30019]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T19:50:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]: dispatch 2024-09-20T19:50:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:06.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]': finished 2024-09-20T19:50:06.114 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[23692]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:50:06.114 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]: dispatch 2024-09-20T19:50:06.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]': finished 2024-09-20T19:50:06.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[30019]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:50:06.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:05 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]: dispatch 2024-09-20T19:50:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:05 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/679611110' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/2187783388"}]': finished 2024-09-20T19:50:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:05 smithi196 ceph-mon[25662]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T19:50:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:05 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]: dispatch 2024-09-20T19:50:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:06 smithi196 ceph-mon[25662]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 168 op/s 2024-09-20T19:50:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:06 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]': finished 2024-09-20T19:50:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:06 smithi196 ceph-mon[25662]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:50:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:06 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]: dispatch 2024-09-20T19:50:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[23692]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 168 op/s 2024-09-20T19:50:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]': finished 2024-09-20T19:50:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[23692]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:50:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]: dispatch 2024-09-20T19:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[30019]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 341 B/s wr, 168 op/s 2024-09-20T19:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/343090257' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/4071369379"}]': finished 2024-09-20T19:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[30019]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T19:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:06 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]: dispatch 2024-09-20T19:50:08.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:07 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]': finished 2024-09-20T19:50:08.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:07 smithi196 ceph-mon[25662]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:50:08.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:07 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2394000116' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:08.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:07 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:08.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]': finished 2024-09-20T19:50:08.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[23692]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:50:08.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2394000116' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:08.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:08.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3634284285' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/375641286"}]': finished 2024-09-20T19:50:08.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[30019]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T19:50:08.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2394000116' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:08.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:07 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]: dispatch 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[23692]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]': finished 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[23692]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]: dispatch 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[30019]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]': finished 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[30019]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:50:09.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:08 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]: dispatch 2024-09-20T19:50:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:08 smithi196 ceph-mon[25662]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-20T19:50:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:08 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1689245221"}]': finished 2024-09-20T19:50:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:08 smithi196 ceph-mon[25662]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T19:50:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:08 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]: dispatch 2024-09-20T19:50:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]': finished 2024-09-20T19:50:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:50:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1609787240' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]': finished 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1609787240' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:09 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2895429437' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1216562938"}]': finished 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1609787240' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]: dispatch 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:09 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]': finished 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]: dispatch 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:10 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]': finished 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]: dispatch 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3680121705"}]': finished 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]: dispatch 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:11.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:10 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]': finished 2024-09-20T19:50:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[30019]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:50:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]: dispatch 2024-09-20T19:50:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]': finished 2024-09-20T19:50:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[23692]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:50:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]: dispatch 2024-09-20T19:50:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:11 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:12.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:12] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.43.0" 2024-09-20T19:50:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:11 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3020028350' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/3616549371"}]': finished 2024-09-20T19:50:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:11 smithi196 ceph-mon[25662]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T19:50:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:11 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]: dispatch 2024-09-20T19:50:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:11 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:12.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[23692]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]': finished 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[23692]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]: dispatch 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[30019]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]': finished 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[30019]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:50:12.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:12 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]: dispatch 2024-09-20T19:50:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:12 smithi196 ceph-mon[25662]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:12 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1450404670' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/930113251"}]': finished 2024-09-20T19:50:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:12 smithi196 ceph-mon[25662]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T19:50:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:12 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]: dispatch 2024-09-20T19:50:14.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]': finished 2024-09-20T19:50:14.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[23692]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:50:14.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]: dispatch 2024-09-20T19:50:14.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]': finished 2024-09-20T19:50:14.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[30019]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:50:14.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:13 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]: dispatch 2024-09-20T19:50:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:13 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1679755085' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1313825268"}]': finished 2024-09-20T19:50:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:13 smithi196 ceph-mon[25662]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T19:50:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:13 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]: dispatch 2024-09-20T19:50:15.012 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:14] ENGINE Bus STOPPING 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:15] ENGINE Bus STOPPED 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]': finished 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/172003469' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:15.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]': finished 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/172003469' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:15.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:15.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:15.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:15] ENGINE Bus STARTING 2024-09-20T19:50:15.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:15] ENGINE Serving on http://:::9283 2024-09-20T19:50:15.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:50:15] ENGINE Bus STARTED 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2511489067' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3685186450"}]': finished 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/172003469' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]: dispatch 2024-09-20T19:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:15.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: Adding iSCSI gateway http://:@172.21.15.184:5000 to Dashboard 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]': finished 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: Adding iSCSI gateway http://:@172.21.15.184:5000 to Dashboard 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:15.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:15.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]': finished 2024-09-20T19:50:15.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:50:15.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:15 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]: dispatch 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: Adding iSCSI gateway http://:@172.21.15.184:5000 to Dashboard 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi184"}]: dispatch 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/3616549371"}]': finished 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T19:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:15 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]: dispatch 2024-09-20T19:50:17.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]': finished 2024-09-20T19:50:17.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[23692]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:50:17.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]: dispatch 2024-09-20T19:50:17.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[23692]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-09-20T19:50:17.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]': finished 2024-09-20T19:50:17.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[30019]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:50:17.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]: dispatch 2024-09-20T19:50:17.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:16 smithi184 ceph-mon[30019]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-09-20T19:50:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:16 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1800011066' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/930113251"}]': finished 2024-09-20T19:50:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:16 smithi196 ceph-mon[25662]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T19:50:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:16 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]: dispatch 2024-09-20T19:50:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:16 smithi196 ceph-mon[25662]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 7 op/s 2024-09-20T19:50:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]': finished 2024-09-20T19:50:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:17 smithi196 ceph-mon[25662]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:50:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:17 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]: dispatch 2024-09-20T19:50:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]': finished 2024-09-20T19:50:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[23692]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:50:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]: dispatch 2024-09-20T19:50:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3621376637' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3643808076"}]': finished 2024-09-20T19:50:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[30019]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T19:50:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:17 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]: dispatch 2024-09-20T19:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]': finished 2024-09-20T19:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 5 op/s 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]': finished 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-09-20T19:50:19.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 5 op/s 2024-09-20T19:50:19.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:19.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:18 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/4115922494' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/1486363706"}]': finished 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 5 op/s 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: Checking dashboard <-> RGW credentials 2024-09-20T19:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:18 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:21 smithi184 ceph-mon[23692]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:21 smithi184 ceph-mon[30019]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:21 smithi196 ceph-mon[25662]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:22] "GET /metrics HTTP/1.1" 200 38437 "" "Prometheus/2.43.0" 2024-09-20T19:50:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:22 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:22 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:22 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[23692]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[30019]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:23 smithi196 ceph-mon[25662]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-09-20T19:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:25.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:25 smithi184 ceph-mon[23692]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s 2024-09-20T19:50:25.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:25 smithi184 ceph-mon[30019]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s 2024-09-20T19:50:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:25 smithi196 ceph-mon[25662]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s 2024-09-20T19:50:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:27 smithi184 ceph-mon[23692]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-09-20T19:50:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:27 smithi184 ceph-mon[30019]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-09-20T19:50:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:27 smithi196 ceph-mon[25662]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-09-20T19:50:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:29 smithi196 ceph-mon[25662]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:29 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:29 smithi184 ceph-mon[23692]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:29 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:29 smithi184 ceph-mon[30019]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:29 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:31 smithi184 ceph-mon[23692]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-09-20T19:50:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:31 smithi184 ceph-mon[30019]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-09-20T19:50:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:31 smithi196 ceph-mon[25662]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-09-20T19:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:32] "GET /metrics HTTP/1.1" 200 38443 "" "Prometheus/2.43.0" 2024-09-20T19:50:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:32 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:32 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:32 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:33 smithi196 ceph-mon[25662]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:33 smithi184 ceph-mon[23692]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:33 smithi184 ceph-mon[30019]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T19:50:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:35 smithi196 ceph-mon[25662]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:35 smithi184 ceph-mon[23692]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:35 smithi184 ceph-mon[30019]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:37 smithi196 ceph-mon[25662]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:37 smithi184 ceph-mon[23692]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:37 smithi184 ceph-mon[30019]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:39.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:39 smithi184 ceph-mon[23692]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:39.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:39 smithi184 ceph-mon[30019]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:39 smithi196 ceph-mon[25662]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:41.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:41 smithi184 ceph-mon[30019]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:41.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:41 smithi184 ceph-mon[23692]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:41 smithi196 ceph-mon[25662]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:42] "GET /metrics HTTP/1.1" 200 38443 "" "Prometheus/2.43.0" 2024-09-20T19:50:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:42 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:42 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:42 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:43 smithi196 ceph-mon[25662]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:43 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:43 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:43 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[23692]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:43.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[30019]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:43.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:43.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:43 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:50:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:50:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:45 smithi196 ceph-mon[25662]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:45 smithi184 ceph-mon[23692]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:45 smithi184 ceph-mon[30019]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:47 smithi196 ceph-mon[25662]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:47 smithi184 ceph-mon[23692]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:47 smithi184 ceph-mon[30019]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:49 smithi196 ceph-mon[25662]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:49 smithi184 ceph-mon[23692]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:49 smithi184 ceph-mon[30019]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:51 smithi184 ceph-mon[23692]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:51 smithi184 ceph-mon[30019]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:51 smithi196 ceph-mon[25662]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:50:52] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:50:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:52 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:52 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:52 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:50:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:53 smithi196 ceph-mon[25662]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:53 smithi184 ceph-mon[23692]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:53 smithi184 ceph-mon[30019]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:55 smithi196 ceph-mon[25662]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:55 smithi184 ceph-mon[23692]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:55 smithi184 ceph-mon[30019]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:57 smithi196 ceph-mon[25662]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:57 smithi184 ceph-mon[23692]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:57 smithi184 ceph-mon[30019]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:50:59.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:59 smithi196 ceph-mon[25662]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:50:59.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:50:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:59 smithi184 ceph-mon[23692]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:50:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:59 smithi184 ceph-mon[30019]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:50:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:01 smithi184 ceph-mon[30019]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:01 smithi184 ceph-mon[23692]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:01 smithi196 ceph-mon[25662]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:02.292 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:02] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:51:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:02 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:02 smithi196 ceph-mon[25662]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:03.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:02 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:03.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:02 smithi184 ceph-mon[23692]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:02 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:02 smithi184 ceph-mon[30019]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:05 smithi184 ceph-mon[30019]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:05 smithi184 ceph-mon[23692]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:05 smithi196 ceph-mon[25662]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:07.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:07 smithi184 ceph-mon[23692]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:07.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:07 smithi184 ceph-mon[30019]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:07 smithi196 ceph-mon[25662]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:09 smithi184 ceph-mon[23692]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:09 smithi184 ceph-mon[30019]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:09 smithi196 ceph-mon[25662]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:11 smithi184 ceph-mon[23692]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:11 smithi184 ceph-mon[30019]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:11 smithi196 ceph-mon[25662]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:12] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:12 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:12 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:12 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:13 smithi184 ceph-mon[23692]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:13 smithi184 ceph-mon[30019]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:13 smithi196 ceph-mon[25662]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:15 smithi184 ceph-mon[23692]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:15 smithi184 ceph-mon[30019]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:15 smithi196 ceph-mon[25662]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:17 smithi184 ceph-mon[30019]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:17 smithi184 ceph-mon[23692]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:17 smithi196 ceph-mon[25662]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:19 smithi196 ceph-mon[25662]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:19 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:19 smithi184 ceph-mon[23692]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:19 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:19 smithi184 ceph-mon[30019]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:19 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:21 smithi196 ceph-mon[25662]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:21 smithi184 ceph-mon[23692]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:21 smithi184 ceph-mon[30019]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:22] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.43.0" 2024-09-20T19:51:22.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:22.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:22.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:22.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:22.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:22.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:22 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:22 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:22 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:22 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[23692]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[30019]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:51:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:23 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:23 smithi196 ceph-mon[25662]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:51:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:51:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:23 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:24 smithi196 ceph-mon[25662]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-20T19:51:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:24 smithi196 ceph-mon[25662]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[23692]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-20T19:51:24.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[23692]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:24.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[30019]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-20T19:51:24.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[30019]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:24.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:24.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:26 smithi196 ceph-mon[25662]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:26 smithi184 ceph-mon[23692]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:26 smithi184 ceph-mon[30019]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:28 smithi196 ceph-mon[25662]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:28 smithi184 ceph-mon[23692]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:28 smithi184 ceph-mon[30019]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:29 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:29 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:29 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:30 smithi196 ceph-mon[25662]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:30 smithi184 ceph-mon[23692]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:30 smithi184 ceph-mon[30019]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-09-20T19:51:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:32] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.43.0" 2024-09-20T19:51:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:32 smithi196 ceph-mon[25662]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:32 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:32 smithi184 ceph-mon[23692]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:32 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:32 smithi184 ceph-mon[30019]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:32 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:34 smithi196 ceph-mon[25662]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:51:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:34 smithi184 ceph-mon[23692]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:51:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:34 smithi184 ceph-mon[30019]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:51:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:36 smithi196 ceph-mon[25662]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:36 smithi184 ceph-mon[23692]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:36 smithi184 ceph-mon[30019]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:38 smithi196 ceph-mon[25662]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:38 smithi184 ceph-mon[23692]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:38 smithi184 ceph-mon[30019]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:40 smithi184 ceph-mon[23692]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:40 smithi184 ceph-mon[30019]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:40 smithi196 ceph-mon[25662]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:42] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.43.0" 2024-09-20T19:51:42.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:42 smithi196 ceph-mon[25662]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:42.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:42 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:42 smithi184 ceph-mon[30019]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:42 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:42 smithi184 ceph-mon[23692]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:42 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:44.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[23692]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:44.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:44.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[30019]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:44.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:44 smithi196 ceph-mon[25662]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:51:45.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-20T19:51:45.726 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:51:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:46 smithi196 ceph-mon[25662]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:46.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:46 smithi184 ceph-mon[23692]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:46.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:46 smithi184 ceph-mon[30019]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:48.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:48 smithi184 ceph-mon[23692]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:48.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:48 smithi184 ceph-mon[30019]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:48.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-20T19:51:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:48 smithi196 ceph-mon[25662]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:48.953 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:51:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:50 smithi196 ceph-mon[25662]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:50 smithi184 ceph-mon[23692]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:50 smithi184 ceph-mon[30019]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:51.351 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-20T19:51:51.550 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:51:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:51:52] "GET /metrics HTTP/1.1" 200 38441 "" "Prometheus/2.43.0" 2024-09-20T19:51:52.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:52 smithi184 ceph-mon[23692]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:52.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:52 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:52.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:52 smithi184 ceph-mon[30019]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:52.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:52 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:52 smithi196 ceph-mon[25662]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:52 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:51:53.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-20T19:51:54.120 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:51:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:54 smithi196 ceph-mon[25662]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:54 smithi184 ceph-mon[23692]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:54 smithi184 ceph-mon[30019]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:55.913 INFO:teuthology.orchestra.run.smithi184.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:51:56.670 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:51:56.679 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi184.front.sepia.ceph.com 2024-09-20T19:51:56.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4b598dd2-7788-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-20T19:51:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:56 smithi196 ceph-mon[25662]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:56 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:56 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:57.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[23692]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:57.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:57.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:57.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[30019]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:51:57.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:51:57.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:56 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:51:57.523 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T19:51:57.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:57 smithi184 ceph-mon[23692]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:51:57.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:57 smithi184 ceph-mon[23692]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:51:57.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:57 smithi184 ceph-mon[30019]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:51:57.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:57 smithi184 ceph-mon[30019]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:51:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:57 smithi196 ceph-mon[25662]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:51:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:57 smithi196 ceph-mon[25662]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:51:58.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:58 smithi184 ceph-mon[23692]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:58.876 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:58 smithi184 ceph-mon[30019]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:58 smithi196 ceph-mon[25662]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:51:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:51:59.857 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:51:59.857 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 error 37s ago 8m - - 2024-09-20T19:51:59.857 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 8m - - 2024-09-20T19:51:59.857 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2m) 37s ago 3m 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3m) 16s ago 3m 90.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (119s) 37s ago 118s 50.6M - 3.8 4404209dfb71 713d74237379 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,8765 running (7m) 16s ago 7m 439M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0715439b5fe2 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (9m) 37s ago 9m 500M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c98817f3cafa 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (9m) 37s ago 9m 51.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (8m) 16s ago 8m 43.9M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (8m) 37s ago 8m 42.3M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3m) 37s ago 3m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3m) 16s ago 3m 15.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7m) 37s ago 7m 56.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7m) 37s ago 7m 60.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6m) 37s ago 6m 54.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6m) 37s ago 6m 48.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:51:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5m) 16s ago 5m 61.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5m) 16s ago 5m 51.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5m) 16s ago 5m 47.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4m) 16s ago 4m 50.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2m) 16s ago 4m 52.7M - 2.43.0 a07b618ecd1d 18bcdb6bdeac 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2m) 37s ago 2m 75.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2m) 16s ago 2m 76.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2m) 37s ago 2m 79.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:51:59.859 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2m) 16s ago 2m 77.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:51:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:51:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:51:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:51:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:52:00.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:52:00.187 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [], 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "", 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "", 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:52:00.465 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:52:00.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[23692]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:00.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[23692]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[23692]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2187132402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:00.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[30019]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:00.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[30019]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[30019]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:00 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2187132402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:00.796 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:52:00.796 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:52:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:52:00.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:00 smithi196 ceph-mon[25662]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:00.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:00 smithi196 ceph-mon[25662]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:00 smithi196 ceph-mon[25662]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:00.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:00 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2187132402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2928521818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2928521818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:52:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:52:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:01 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2928521818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:01 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:02] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:52:02.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:02 smithi196 ceph-mon[25662]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:02.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:02 smithi196 ceph-mon[25662]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-09-20T19:52:02.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:02 smithi196 ceph-mon[25662]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:02.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:02 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:02.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:02 smithi196 ceph-mon[25662]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:52:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[23692]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[23692]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-09-20T19:52:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[23692]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[23692]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[30019]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[30019]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[30019]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:02 smithi184 ceph-mon[30019]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[23692]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[30019]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:04 smithi196 ceph-mon[25662]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:05.675 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:05 smithi184 ceph-mon[23692]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:05.675 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:05 smithi184 ceph-mon[30019]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:05 smithi196 ceph-mon[25662]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:08 smithi184 ceph-mon[23692]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:08 smithi184 ceph-mon[30019]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:08 smithi196 ceph-mon[25662]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:52:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:10 smithi184 ceph-mon[23692]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-09-20T19:52:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:10 smithi184 ceph-mon[30019]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-09-20T19:52:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:10 smithi196 ceph-mon[25662]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-09-20T19:52:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:12] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:52:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:12 smithi184 ceph-mon[23692]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:52:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:12 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:12 smithi184 ceph-mon[30019]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:52:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:12 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:12 smithi196 ceph-mon[25662]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:52:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:12 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:14 smithi184 ceph-mon[23692]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:14 smithi184 ceph-mon[30019]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:14 smithi196 ceph-mon[25662]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:16 smithi184 ceph-mon[23692]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:16 smithi184 ceph-mon[30019]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:16 smithi196 ceph-mon[25662]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:18 smithi184 ceph-mon[23692]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:18 smithi184 ceph-mon[30019]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:18.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:18 smithi196 ceph-mon[25662]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:20 smithi184 ceph-mon[23692]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:20 smithi184 ceph-mon[30019]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:20.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:20 smithi196 ceph-mon[25662]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:22] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.43.0" 2024-09-20T19:52:22.593 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:22 smithi196 ceph-mon[25662]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:22.593 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:22 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:22 smithi184 ceph-mon[23692]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:22 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:22 smithi184 ceph-mon[30019]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:22 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[23692]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[30019]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:24 smithi196 ceph-mon[25662]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:26 smithi184 ceph-mon[30019]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:26 smithi184 ceph-mon[23692]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:26 smithi196 ceph-mon[25662]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:28 smithi184 ceph-mon[23692]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:28 smithi184 ceph-mon[30019]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:28 smithi196 ceph-mon[25662]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:29 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:29 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:29 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:30 smithi184 ceph-mon[23692]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:30 smithi184 ceph-mon[30019]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:30 smithi196 ceph-mon[25662]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:31.122 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:52:31.736 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:52:31.736 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 error 30s ago 9m - - 2024-09-20T19:52:31.736 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 9m - - 2024-09-20T19:52:31.736 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3m) 30s ago 4m 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4m) 7s ago 4m 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2m) 30s ago 2m 50.9M - 3.8 4404209dfb71 713d74237379 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,8765 running (8m) 7s ago 8m 439M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0715439b5fe2 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (10m) 30s ago 10m 501M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c98817f3cafa 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (10m) 30s ago 10m 52.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (8m) 7s ago 8m 41.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (8m) 30s ago 8m 43.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4m) 30s ago 4m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4m) 7s ago 4m 15.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (8m) 30s ago 8m 56.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7m) 30s ago 7m 60.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7m) 30s ago 7m 54.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6m) 30s ago 6m 48.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6m) 7s ago 6m 62.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:52:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6m) 7s ago 6m 51.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5m) 7s ago 5m 47.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5m) 7s ago 5m 50.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3m) 7s ago 4m 54.9M - 2.43.0 a07b618ecd1d 18bcdb6bdeac 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3m) 30s ago 3m 75.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3m) 7s ago 3m 76.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2m) 30s ago 2m 80.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:52:31.738 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2m) 7s ago 2m 78.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:52:32.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:52:32.099 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:52:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:32] "GET /metrics HTTP/1.1" 200 38442 "" "Prometheus/2.43.0" 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [], 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "", 2024-09-20T19:52:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 image", 2024-09-20T19:52:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:52:32.409 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:52:32.764 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:52:32.764 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:52:32.764 INFO:teuthology.orchestra.run.smithi184.stdout: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[23692]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[23692]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[23692]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3578901890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[30019]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[30019]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[30019]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:32.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:32.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:32 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3578901890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:32 smithi196 ceph-mon[25662]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:32 smithi196 ceph-mon[25662]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:32 smithi196 ceph-mon[25662]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:32 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:32 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3578901890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:52:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:33 smithi196 ceph-mon[25662]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:33 smithi196 ceph-mon[25662]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:33 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/4048058132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[30019]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[30019]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/4048058132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[23692]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[23692]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:52:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:33 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4048058132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:52:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:34 smithi196 ceph-mon[25662]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:34 smithi184 ceph-mon[23692]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:34 smithi184 ceph-mon[30019]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:36 smithi196 ceph-mon[25662]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:36 smithi184 ceph-mon[30019]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:36 smithi184 ceph-mon[23692]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:38 smithi196 ceph-mon[25662]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:38 smithi184 ceph-mon[23692]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:38 smithi184 ceph-mon[30019]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:40.849 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:40 smithi184 ceph-mon[23692]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:40.849 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:40 smithi184 ceph-mon[30019]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:40 smithi196 ceph-mon[25662]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:42] "GET /metrics HTTP/1.1" 200 38442 "" "Prometheus/2.43.0" 2024-09-20T19:52:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:42 smithi196 ceph-mon[25662]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:42 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:42 smithi184 ceph-mon[23692]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:42 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:42 smithi184 ceph-mon[30019]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:42 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[23692]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[30019]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:44.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:44 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:44 smithi196 ceph-mon[25662]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:44 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[23692]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[30019]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:46.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:46.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:52:46.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:46 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:46 smithi196 ceph-mon[25662]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:52:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:46 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:52:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[23692]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T19:52:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[23692]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T19:52:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[23692]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T19:52:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[23692]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi196 2024-09-20T19:52:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[30019]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T19:52:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[30019]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T19:52:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[30019]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T19:52:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:47 smithi184 ceph-mon[30019]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi196 2024-09-20T19:52:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:47 smithi196 ceph-mon[25662]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T19:52:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:47 smithi196 ceph-mon[25662]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T19:52:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:47 smithi196 ceph-mon[25662]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T19:52:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:47 smithi196 ceph-mon[25662]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi196 2024-09-20T19:52:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:48 smithi196 ceph-mon[25662]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:48 smithi184 ceph-mon[30019]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:48 smithi184 ceph-mon[23692]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:50 smithi196 ceph-mon[25662]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:50 smithi184 ceph-mon[23692]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:50 smithi184 ceph-mon[30019]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:52:52] "GET /metrics HTTP/1.1" 200 38442 "" "Prometheus/2.43.0" 2024-09-20T19:52:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:52 smithi184 ceph-mon[23692]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:52 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:52 smithi184 ceph-mon[30019]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:52 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:52 smithi196 ceph-mon[25662]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:52 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:52:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:54 smithi196 ceph-mon[25662]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:54 smithi184 ceph-mon[23692]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:54 smithi184 ceph-mon[30019]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:56 smithi196 ceph-mon[25662]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:56 smithi184 ceph-mon[23692]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:56 smithi184 ceph-mon[30019]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:52:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:58 smithi196 ceph-mon[25662]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:58 smithi184 ceph-mon[23692]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:58 smithi184 ceph-mon[30019]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:52:59 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:52:59 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:52:59 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:00 smithi196 ceph-mon[25662]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:00 smithi184 ceph-mon[23692]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:00 smithi184 ceph-mon[30019]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:02] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:53:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:02 smithi196 ceph-mon[25662]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:02 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:02 smithi184 ceph-mon[23692]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:02 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:02 smithi184 ceph-mon[30019]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:02 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:03.059 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:53:03.592 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:53:03.592 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 error 62s ago 9m - - 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 9m - - 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3m) 62s ago 5m 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4m) 19s ago 4m 90.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3m) 62s ago 3m 50.9M - 3.8 4404209dfb71 713d74237379 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,8765 running (8m) 19s ago 8m 440M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0715439b5fe2 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (10m) 62s ago 10m 501M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c98817f3cafa 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (10m) 62s ago 10m 52.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (9m) 19s ago 9m 41.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (9m) 62s ago 9m 43.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4m) 62s ago 4m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4m) 19s ago 4m 15.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (8m) 62s ago 8m 56.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:53:03.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (8m) 62s ago 8m 60.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7m) 62s ago 7m 54.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7m) 62s ago 7m 48.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7m) 19s ago 7m 62.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6m) 19s ago 6m 51.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6m) 19s ago 6m 48.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5m) 19s ago 5m 50.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3m) 19s ago 5m 54.9M - 2.43.0 a07b618ecd1d 18bcdb6bdeac 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3m) 62s ago 3m 75.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3m) 19s ago 3m 76.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3m) 62s ago 3m 80.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:53:03.594 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3m) 19s ago 3m 78.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:53:03.911 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:53:03.911 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:53:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T19:53:03.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:53:03.913 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [], 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "0/23 daemons upgraded", 2024-09-20T19:53:04.188 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d image on host smithi196", 2024-09-20T19:53:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:53:04.189 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:53:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:53:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T19:53:04.503 INFO:teuthology.orchestra.run.smithi184.stdout: Cephadm agent on host smithi184 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T19:53:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3072489951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1109151556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3072489951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1109151556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:04.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3072489951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:04.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1109151556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:05 smithi184 ceph-mon[23692]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:05 smithi184 ceph-mon[30019]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:05 smithi196 ceph-mon[25662]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:06 smithi196 ceph-mon[25662]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:06 smithi184 ceph-mon[23692]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:06 smithi184 ceph-mon[30019]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:08 smithi196 ceph-mon[25662]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:08.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:08 smithi184 ceph-mon[23692]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:08.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:08 smithi184 ceph-mon[30019]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:10 smithi196 ceph-mon[25662]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:10 smithi184 ceph-mon[23692]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:10 smithi184 ceph-mon[30019]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:12] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.43.0" 2024-09-20T19:53:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:12 smithi196 ceph-mon[25662]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:12 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:12 smithi184 ceph-mon[23692]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:12 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:12 smithi184 ceph-mon[30019]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:12 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:14 smithi196 ceph-mon[25662]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:14 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:14 smithi184 ceph-mon[23692]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:14 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:14 smithi184 ceph-mon[30019]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:14 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:16 smithi196 ceph-mon[25662]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:16 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:16 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:16 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:16 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[30019]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[23692]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:16 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:17 smithi196 ceph-mon[25662]: Upgrade: Updating mgr.x 2024-09-20T19:53:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:17 smithi196 ceph-mon[25662]: Deploying daemon mgr.x on smithi196 2024-09-20T19:53:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:17 smithi184 ceph-mon[23692]: Upgrade: Updating mgr.x 2024-09-20T19:53:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:17 smithi184 ceph-mon[23692]: Deploying daemon mgr.x on smithi196 2024-09-20T19:53:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:17 smithi184 ceph-mon[30019]: Upgrade: Updating mgr.x 2024-09-20T19:53:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:17 smithi184 ceph-mon[30019]: Deploying daemon mgr.x on smithi196 2024-09-20T19:53:18.210 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:17 smithi196 systemd[1]: Stopping Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:53:18.490 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:18 smithi196 podman[55995]: 2024-09-20 19:53:18.209351213 +0000 UTC m=+0.106920590 container died 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T19:53:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:18 smithi184 ceph-mon[23692]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:18 smithi184 ceph-mon[30019]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:18 smithi196 ceph-mon[25662]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:18.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:18 smithi196 podman[55995]: 2024-09-20 19:53:18.571873871 +0000 UTC m=+0.469443247 container cleanup 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=reef-0b2cc4d) 2024-09-20T19:53:18.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:18 smithi196 bash[55995]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x 2024-09-20T19:53:19.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:18 smithi196 podman[56010]: 2024-09-20 19:53:18.889952428 +0000 UTC m=+0.671504358 container remove 0715439b5fe2334aadd5bbb0c2670a0ae9df73ed1b48bb92348dc310f714f02f (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-20T19:53:19.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:18 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:53:19.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T19:53:19.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 systemd[1]: Stopped Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:53:19.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Consumed 11.286s CPU time. 2024-09-20T19:53:19.458 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 systemd[1]: Starting Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:53:19.744 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 podman[56135]: 2024-09-20 19:53:19.45783026 +0000 UTC m=+0.105374018 container create 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T19:53:19.744 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 podman[56135]: 2024-09-20 19:53:19.369469544 +0000 UTC m=+0.017013301 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:53:19.744 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 podman[56135]: 2024-09-20 19:53:19.660084223 +0000 UTC m=+0.307627986 container init 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T19:53:19.744 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 podman[56135]: 2024-09-20 19:53:19.663091937 +0000 UTC m=+0.310635696 container start 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T19:53:19.745 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 ceph-mgr[56169]: -- 172.21.15.196:0/1639512443 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5622717ef0e0 con 0x5622717cd400 2024-09-20T19:53:20.045 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 bash[56135]: 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd 2024-09-20T19:53:20.045 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:19 smithi196 systemd[1]: Started Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:53:20.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:20.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:20.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:20 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:20.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:20.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:20 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:22] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.43.0" 2024-09-20T19:53:22.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:22 smithi184 ceph-mon[23692]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:22.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:22 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:22.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:22 smithi196 ceph-mon[25662]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:22.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:22 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:22 smithi184 ceph-mon[30019]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:22 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:23.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:24.412 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.063+0000 7f6b7ed03640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T19:53:24.413 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setuser ceph since I am not root 2024-09-20T19:53:24.413 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ignoring --setgroup ceph since I am not root 2024-09-20T19:53:24.413 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.207+0000 7fe6c8fd8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T19:53:24.413 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.264+0000 7fe6c8fd8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T19:53:24.413 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.411+0000 7fe6c8fd8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.477+0000 7fe6c8fd8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.627+0000 7fe6c8fd8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: Activating manager daemon x 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[23692]: mgrmap e22: x(active, starting, since 0.00639005s) 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:24.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: Activating manager daemon x 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T19:53:24.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-mon[30019]: mgrmap e22: x(active, starting, since 0.00639005s) 2024-09-20T19:53:24.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:53:24.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: Activating manager daemon x 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: from='mgr.24338 172.21.15.184:0/144225937' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-mon[25662]: mgrmap e22: x(active, starting, since 0.00639005s) 2024-09-20T19:53:24.983 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.797+0000 7f979b708040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.797+0000 7f979b708040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ^ 2024-09-20T19:53:24.984 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: SyntaxError: EOL while scanning string literal 2024-09-20T19:53:24.985 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.985 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.798+0000 7f979b708040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:53:24.985 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.798+0000 7f979b708040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:53:24.985 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:24.980+0000 7f979b708040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:53:25.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:24.776+0000 7fe6c8fd8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:24] ENGINE Bus STARTING 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: CherryPy Checker: 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: The Application mounted at '' has an empty config. 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:25] ENGINE Serving on http://:::9283 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:25] ENGINE Bus STARTED 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:25] ENGINE Bus STOPPING 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:53:25.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:25] ENGINE Bus STOPPED 2024-09-20T19:53:25.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:25.097+0000 7fe6c8fd8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T19:53:25.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:25.158+0000 7fe6c8fd8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T19:53:25.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:25.343+0000 7fe6c8fd8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T19:53:25.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: Active manager daemon x restarted 2024-09-20T19:53:25.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: Activating manager daemon x 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: mgrmap e23: x(active, starting, since 0.0116177s) 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:53:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: Manager daemon x is now available 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:25.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:25.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: Active manager daemon x restarted 2024-09-20T19:53:25.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: Activating manager daemon x 2024-09-20T19:53:25.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:53:25.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: mgrmap e23: x(active, starting, since 0.0116177s) 2024-09-20T19:53:25.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:53:25.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: Manager daemon x is now available 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: Active manager daemon x restarted 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: Activating manager daemon x 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: mgrmap e23: x(active, starting, since 0.0116177s) 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:53:25.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: Manager daemon x is now available 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:25.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:26.121 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:26] ENGINE Bus STARTING 2024-09-20T19:53:26.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:26] ENGINE Serving on http://:::9283 2024-09-20T19:53:26.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:53:26] ENGINE Bus STARTED 2024-09-20T19:53:26.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:26.291+0000 7fe6c8fd8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T19:53:26.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:26.360+0000 7fe6c8fd8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T19:53:26.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Checking for cert/key for grafana.a 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Migrating grafana.a cert to cert store 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: Migrating grafana.a key to cert store 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:26.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:26.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:26 smithi196 ceph-mon[25662]: mgrmap e24: x(active, since 1.01445s) 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Checking for cert/key for grafana.a 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Migrating grafana.a cert to cert store 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: Migrating grafana.a key to cert store 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:27.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[23692]: mgrmap e24: x(active, since 1.01445s) 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Checking for cert/key for grafana.a 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Migrating grafana.a cert to cert store 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: Migrating grafana.a key to cert store 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:27.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T19:53:27.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:26 smithi184 ceph-mon[30019]: mgrmap e24: x(active, since 1.01445s) 2024-09-20T19:53:27.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.122+0000 7fe6c8fd8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T19:53:27.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.238+0000 7fe6c8fd8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:27.082+0000 7f96febdd640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:53:27] ENGINE Error in HTTPServer.serve 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self._connections.run(self.expiration_interval) 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self._run(expiration_interval) 2024-09-20T19:53:27.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: s = self.context.wrap_socket( 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.sslsocket_class._create( 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self.do_handshake() 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self._sslobj.do_handshake() 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:53:27.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:27.639 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.384+0000 7fe6c8fd8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T19:53:27.639 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.449+0000 7fe6c8fd8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T19:53:27.639 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.605+0000 7fe6c8fd8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T19:53:27.890 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.681+0000 7fe6c8fd8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T19:53:27.890 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.746+0000 7fe6c8fd8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T19:53:27.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:27.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:27.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: Reconfiguring daemon agent.smithi196 on smithi196 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: Deploying cephadm binary to smithi184 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: Reconfiguring daemon agent.smithi196 on smithi196 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: Deploying cephadm binary to smithi184 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T19:53:27.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:53:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: Reconfiguring daemon agent.smithi184 on smithi184 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: Reconfiguring daemon agent.smithi196 on smithi196 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: Deploying cephadm binary to smithi184 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T19:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:27 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:53:28.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:27.958+0000 7fe6c8fd8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T19:53:28.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:28.036+0000 7fe6c8fd8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T19:53:28.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:28.104+0000 7fe6c8fd8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T19:53:28.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:28.189+0000 7fe6c8fd8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T19:53:28.450 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:28.254+0000 7fe6c8fd8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T19:53:28.450 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:28.318+0000 7fe6c8fd8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: [20/Sep/2024:19:53:26] ENGINE Bus STARTING 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: [20/Sep/2024:19:53:27] ENGINE Serving on https://172.21.15.196:7150 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: [20/Sep/2024:19:53:27] ENGINE Error in HTTPServer.serve 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:53:28.773 INFO:journalctl@ceph.mon.c.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.do_handshake() 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: [20/Sep/2024:19:53:27] ENGINE Serving on http://172.21.15.196:8765 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: [20/Sep/2024:19:53:27] ENGINE Bus STARTED 2024-09-20T19:53:28.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: Deploying cephadm binary to smithi196 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: mgrmap e25: x(active, since 2s) 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: [20/Sep/2024:19:53:26] ENGINE Bus STARTING 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:28.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: [20/Sep/2024:19:53:27] ENGINE Serving on https://172.21.15.196:7150 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: [20/Sep/2024:19:53:27] ENGINE Error in HTTPServer.serve 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:53:28.776 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: [20/Sep/2024:19:53:27] ENGINE Serving on http://172.21.15.196:8765 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: [20/Sep/2024:19:53:27] ENGINE Bus STARTED 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: Deploying cephadm binary to smithi196 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: mgrmap e25: x(active, since 2s) 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:28.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:28.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: [20/Sep/2024:19:53:26] ENGINE Bus STARTING 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: [20/Sep/2024:19:53:27] ENGINE Serving on https://172.21.15.196:7150 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: [20/Sep/2024:19:53:27] ENGINE Error in HTTPServer.serve 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:53:28.906 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: [20/Sep/2024:19:53:27] ENGINE Serving on http://172.21.15.196:8765 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: [20/Sep/2024:19:53:27] ENGINE Bus STARTED 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: Deploying cephadm binary to smithi196 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: mgrmap e25: x(active, since 2s) 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.907 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:28.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:29.436 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:29.127+0000 7fe6c8fd8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:53:29] ENGINE Bus STARTING 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: CherryPy Checker: 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: The Application mounted at '' has an empty config. 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:53:29] ENGINE Serving on http://:::9283 2024-09-20T19:53:29.437 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: [20/Sep/2024:19:53:29] ENGINE Bus STARTED 2024-09-20T19:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[23692]: Standby manager daemon y started 2024-09-20T19:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T19:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[23692]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[30019]: Standby manager daemon y started 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T19:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:29 smithi184 ceph-mon[30019]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:53:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:29 smithi196 ceph-mon[25662]: Standby manager daemon y started 2024-09-20T19:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:29 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T19:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:29 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:29 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T19:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:29 smithi196 ceph-mon[25662]: from='mgr.? 172.21.15.184:0/483131048' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:53:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:30 smithi196 ceph-mon[25662]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:53:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:30 smithi196 ceph-mon[25662]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-20T19:53:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[23692]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:53:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[23692]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-20T19:53:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[30019]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:53:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:30 smithi184 ceph-mon[30019]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-20T19:53:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T19:53:32.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:32 smithi184 ceph-mon[23692]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:32.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:32 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:32.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:32 smithi184 ceph-mon[30019]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:32.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:32 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:32 smithi196 ceph-mon[25662]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:32 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:33.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:33.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:34.803 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:53:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:34 smithi184 ceph-mon[23692]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:53:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:34 smithi184 ceph-mon[30019]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:53:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:34 smithi196 ceph-mon[25662]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:53:35.345 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 starting - - - - 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 starting - - - - 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4m) 11s ago 5m 27.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5m) 31s ago 5m 92.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3m) 11s ago 3m 51.2M - 3.8 4404209dfb71 713d74237379 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,9283,8765 starting - - - - 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (11m) 11s ago 11m 503M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c98817f3cafa 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (11m) 11s ago 11m 52.6M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (9m) 31s ago 9m 41.3M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (9m) 11s ago 9m 40.6M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5m) 11s ago 5m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5m) 31s ago 5m 15.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (9m) 11s ago 9m 57.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (8m) 11s ago 8m 61.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (8m) 11s ago 8m 55.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7m) 11s ago 7m 49.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7m) 31s ago 7m 62.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7m) 31s ago 7m 51.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6m) 31s ago 6m 48.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6m) 31s ago 6m 50.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4m) 31s ago 5m 54.9M - 2.43.0 a07b618ecd1d 18bcdb6bdeac 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4m) 11s ago 4m 76.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4m) 31s ago 4m 76.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3m) 11s ago 3m 80.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:53:35.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3m) 31s ago 3m 78.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T19:53:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 16, 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:53:35.665 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:53:35.957 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:53:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:53:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:53:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:53:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [], 2024-09-20T19:53:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "", 2024-09-20T19:53:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "", 2024-09-20T19:53:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:53:35.958 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:53:35.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:35 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/604651811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:35.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:35 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/604651811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:35 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/604651811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:53:36.280 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:53:36.280 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:53:36.281 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:53:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:53:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1214900663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1214900663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1214900663' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:38.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Adjusting osd_memory_target on smithi184 to 2480M 2024-09-20T19:53:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:38.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[23692]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[23692]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[30019]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:39.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:39 smithi184 ceph-mon[30019]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:53:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:39 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:53:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:39 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:53:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:39 smithi196 ceph-mon[25662]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:39 smithi196 ceph-mon[25662]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: Deploying daemon agent.smithi184 on smithi184 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: Deploying daemon agent.smithi184 on smithi184 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:40.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:40.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:40 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: Deploying daemon agent.smithi184 on smithi184 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:40 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[23692]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:41.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:53:41.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:53:41.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[30019]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:41.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:41.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:41 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:41 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:53:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:41 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:53:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:41 smithi196 ceph-mon[25662]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:41 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:41 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:42.220 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T19:53:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:42 smithi184 ceph-mon[23692]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:42 smithi184 ceph-mon[30019]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:42 smithi196 ceph-mon[25662]: from='client.14919 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:53:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:43 smithi196 ceph-mon[25662]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:43.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:43 smithi184 ceph-mon[23692]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:43.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:43 smithi184 ceph-mon[30019]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:53:43.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:43.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:43.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:43.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:45.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:45 smithi184 ceph-mon[23692]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-20T19:53:45.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:45 smithi184 ceph-mon[30019]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-20T19:53:45.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-mon[25662]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:53:45.324+0000 7f972ac35640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Creating ceph-iscsi config... 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:45.609 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return _run_code(code, main_globals, None, 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: exec(code, run_globals) 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:45.610 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.wait_async( 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return future.result(timeout) 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:53:45.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.__get_result() 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: raise self._exception 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: out, err, code = await self._run_cephadm( 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: raise OrchestratorError( 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Creating ceph-iscsi config... 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:45.612 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return _run_code(code, main_globals, None, 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: exec(code, run_globals) 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:45.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:45.614 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:53:45 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:46 smithi196 ceph-mon[25662]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.wait_async( 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-09-20T19:53:46.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise OrchestratorError( 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:46 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:46.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:46 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[23692]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:53:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.wait_async( 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: return future.result(timeout) 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[30019]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.wait_async( 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:53:46.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: return future.result(timeout) 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.__get_result() 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise self._exception 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.554 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.555 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.__get_result() 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise self._exception 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:53:46.556 INFO:journalctl@ceph.mon.c.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:53:46.557 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:46.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:46.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:46 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:53:47.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:47 smithi196 ceph-mon[25662]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:53:47.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:47 smithi196 ceph-mon[25662]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:47.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:47 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:47.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:47 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:47.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[23692]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:53:47.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[23692]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:47.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:47.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:47.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[30019]: Deploying daemon agent.smithi196 on smithi196 2024-09-20T19:53:47.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[30019]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:53:47.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:47.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:47 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: Detected new or changed devices on smithi184 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:48 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: Detected new or changed devices on smithi184 2024-09-20T19:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: Detected new or changed devices on smithi184 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:48 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:49.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:49 smithi196 ceph-mon[25662]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-09-20T19:53:49.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:49 smithi196 ceph-mon[25662]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:49.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:49 smithi196 ceph-mon[25662]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[23692]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-09-20T19:53:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[23692]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[23692]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[30019]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-09-20T19:53:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[30019]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:49 smithi184 ceph-mon[30019]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:51 smithi196 ceph-mon[25662]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:51.890 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:51 smithi196 systemd[1]: Stopping Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:53:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:51 smithi184 ceph-mon[23692]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:51 smithi184 ceph-mon[30019]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T19:53:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[23920]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:53:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T19:53:52.796 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:53:52.796 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:53:52.796 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:53:52.796 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:53:52.796 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[49215]: ts=2024-09-20T19:53:52.468Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T19:53:52.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 podman[59209]: 2024-09-20 19:53:52.478537841 +0000 UTC m=+0.148151106 container died 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:53.112 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 podman[59209]: 2024-09-20 19:53:52.893159587 +0000 UTC m=+0.562772851 container cleanup 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:53.112 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:52 smithi196 bash[59209]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:53:53.377 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 podman[59233]: 2024-09-20 19:53:53.110987572 +0000 UTC m=+0.634938560 container remove 18bcdb6bdeacc33061ca271890387504c85cd74b1426f01d3d277b687b41aecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:53.700 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:53 smithi196 ceph-mon[25662]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:53.700 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T19:53:53.701 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 systemd[1]: Stopped Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:53:53.701 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.843s CPU time. 2024-09-20T19:53:53.701 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 systemd[1]: Starting Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:53:53.701 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 podman[59419]: 2024-09-20 19:53:53.603384715 +0000 UTC m=+0.097962943 container create 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:53.701 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:53 smithi196 podman[59419]: 2024-09-20 19:53:53.523413883 +0000 UTC m=+0.017992126 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T19:53:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:53 smithi184 ceph-mon[23692]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:53 smithi184 ceph-mon[30019]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:53.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:53:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:53:53.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 podman[59419]: 2024-09-20 19:53:54.442317303 +0000 UTC m=+0.936895541 container init 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 podman[59419]: 2024-09-20 19:53:54.445478893 +0000 UTC m=+0.940057129 container start 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.468Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.468Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.468Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.468Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.468Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.470Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:53:54.729 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.470Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.472Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.473Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.473Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.473Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.48µs 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.473Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.474Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.482Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.483Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.483Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.822µs wal_replay_duration=9.27873ms wbl_replay_duration=117ns total_replay_duration=9.316308ms 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.484Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.484Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.484Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:53:54.730 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.505Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.852263ms db_storage=678ns remote_storage=1.617µs web_handler=413ns query_engine=540ns scrape=452.894µs scrape_sd=152.495µs notify=25.686µs notify_sd=29.724µs rules=19.816682ms tracing=6.99µs 2024-09-20T19:53:54.731 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.505Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T19:53:54.731 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:53:54.505Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:53:54.731 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 bash[59419]: 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 2024-09-20T19:53:54.731 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:53:54 smithi196 systemd[1]: Started Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:53:55.828 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:55.828 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:55.828 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:55.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:53:55.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:55.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:55 smithi196 ceph-mon[25662]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:53:56.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:56.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[23692]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 305 B/s rd, 0 op/s 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:53:56.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:53:56.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:55 smithi184 ceph-mon[30019]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: Upgrade: Updating mgr.y 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[30019]: Deploying daemon mgr.y on smithi184 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: Upgrade: Updating mgr.y 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:56.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:56 smithi184 ceph-mon[23692]: Deploying daemon mgr.y on smithi184 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: Upgrade: Updating mgr.y 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:53:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:53:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:53:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:56 smithi196 ceph-mon[25662]: Deploying daemon mgr.y on smithi184 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[23692]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state', 'daemon prometheus.a on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:57 smithi184 ceph-mon[30019]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state', 'daemon prometheus.a on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:53:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:57 smithi196 ceph-mon[25662]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state', 'daemon prometheus.a on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:53:58.683 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:58 smithi184 systemd[1]: Stopping Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:53:58.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:58 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2829783560' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:53:58.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:58 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]: dispatch 2024-09-20T19:53:58.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:58 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2829783560' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:53:58.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:58 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]: dispatch 2024-09-20T19:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:58 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2829783560' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:58 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]: dispatch 2024-09-20T19:53:59.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:58 smithi184 podman[83597]: 2024-09-20 19:53:58.997990054 +0000 UTC m=+0.153661460 container died c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T19:53:59.744 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 podman[83597]: 2024-09-20 19:53:59.508926417 +0000 UTC m=+0.664597858 container cleanup c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-09-20T19:53:59.745 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 bash[83597]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y 2024-09-20T19:53:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:59 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]': finished 2024-09-20T19:53:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:59 smithi196 ceph-mon[25662]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:53:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:59 smithi196 ceph-mon[25662]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:53:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:53:59 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]: dispatch 2024-09-20T19:54:00.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]': finished 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[23692]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[23692]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]: dispatch 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 podman[83628]: 2024-09-20 19:53:59.743998404 +0000 UTC m=+0.740365884 container remove c98817f3cafa6ad2fe8dd1cc22f0a8b1213b9e9adcc843885949f9d59888d5c9 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 systemd[1]: Stopped Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:00.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:53:59 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Consumed 1min 926ms CPU time. 2024-09-20T19:54:00.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2938660213' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3132291814"}]': finished 2024-09-20T19:54:00.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[30019]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T19:54:00.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[30019]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:00.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:53:59 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]: dispatch 2024-09-20T19:54:00.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 systemd[1]: Starting Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:00.600 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 podman[83834]: 2024-09-20 19:54:00.255038926 +0000 UTC m=+0.017454234 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:54:00.600 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 podman[83834]: 2024-09-20 19:54:00.378941162 +0000 UTC m=+0.141356469 container create c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-20T19:54:00.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]': finished 2024-09-20T19:54:00.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[23692]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:54:00.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2161375968' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]': finished 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[30019]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2161375968' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 podman[83834]: 2024-09-20 19:54:00.624066031 +0000 UTC m=+0.386481354 container init c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0) 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 podman[83834]: 2024-09-20 19:54:00.627130593 +0000 UTC m=+0.389545904 container start c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T19:54:00.934 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 ceph-mgr[83905]: -- 172.21.15.184:0/2790738927 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f76171f0e0 con 0x55f7616fd000 2024-09-20T19:54:00.935 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 bash[83834]: c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f 2024-09-20T19:54:00.935 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:00 smithi184 systemd[1]: Started Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:00 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2237033665' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6801/1203091601"}]': finished 2024-09-20T19:54:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:00 smithi196 ceph-mon[25662]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T19:54:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:00 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2161375968' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:00 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]: dispatch 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]': finished 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/789532802' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 1 op/s 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]': finished 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/789532802' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 1 op/s 2024-09-20T19:54:02.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:01 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:6800/1203091601"}]': finished 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/789532802' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]: dispatch 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 1 op/s 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:02.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:02.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:02.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:01 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]': finished 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[23692]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1788879408' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]': finished 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[30019]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:54:03.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:54:03.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1788879408' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:02 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:02 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/3901329294"}]': finished 2024-09-20T19:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:02 smithi196 ceph-mon[25662]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T19:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:02 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T19:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:02 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1788879408' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:02 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]: dispatch 2024-09-20T19:54:03.666 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:03.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:03.666 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:03.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-mon[25662]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]': finished 2024-09-20T19:54:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-mon[25662]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:54:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/595477953' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[23692]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]': finished 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[23692]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/595477953' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[30019]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/265488425"}]': finished 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[30019]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T19:54:04.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/595477953' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:04.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:04 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]: dispatch 2024-09-20T19:54:05.110 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:04 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:54:04] "GET /metrics HTTP/1.1" 200 38661 "" "Prometheus/2.43.0" 2024-09-20T19:54:05.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]': finished 2024-09-20T19:54:05.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-mon[23692]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:54:05.344 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]': finished 2024-09-20T19:54:05.344 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-mon[30019]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:54:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:05 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/129943563"}]': finished 2024-09-20T19:54:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:05 smithi196 ceph-mon[25662]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T19:54:05.958 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.774+0000 7fc663148040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.774+0000 7fc663148040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:54:05.959 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: ^ 2024-09-20T19:54:05.960 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: SyntaxError: EOL while scanning string literal 2024-09-20T19:54:05.960 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.960 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.775+0000 7fc663148040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:54:05.960 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.775+0000 7fc663148040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:54:05.960 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:05.956+0000 7fc663148040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[23692]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[23692]: Standby manager daemon y restarted 2024-09-20T19:54:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[23692]: Standby manager daemon y started 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[30019]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[30019]: Standby manager daemon y restarted 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-mon[30019]: Standby manager daemon y started 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: [20/Sep/2024:19:54:05] ENGINE Bus STARTING 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: CherryPy Checker: 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: The Application mounted at '' has an empty config. 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:05 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: 2024-09-20T19:54:06.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: [20/Sep/2024:19:54:06] ENGINE Serving on http://:::9283 2024-09-20T19:54:06.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:06 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[83888]: [20/Sep/2024:19:54:06] ENGINE Bus STARTED 2024-09-20T19:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-mon[25662]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-mon[25662]: Standby manager daemon y restarted 2024-09-20T19:54:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-mon[25662]: Standby manager daemon y started 2024-09-20T19:54:06.572 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:54:06.900 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:06.896+0000 7f972ac35640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:06.900 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Creating ceph-iscsi config... 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 10m - - 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 10m - - 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5m) 19s ago 6m 27.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5m) 10s ago 5m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 unknown 19s ago 4m - - 2024-09-20T19:54:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,9283,8765 running (47s) 10s ago 9m 489M - 19.3.0-5127-ge8baca1e 82f5f45939a4 0f49b238ddd4 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 starting - - - - 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (11m) 19s ago 11m 56.6M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (10m) 10s ago 10m 47.5M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (10m) 19s ago 10m 44.7M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5m) 19s ago 5m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5m) 10s ago 5m 16.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (9m) 19s ago 9m 58.1M 2480M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (9m) 19s ago 9m 61.7M 2480M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (8m) 19s ago 8m 55.7M 2480M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (8m) 19s ago 8m 50.0M 2480M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (8m) 10s ago 8m 63.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7m) 10s ago 7m 52.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7m) 10s ago 7m 48.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6m) 10s ago 6m 51.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:54:07.167 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 unknown 10s ago 6m - - 2024-09-20T19:54:07.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4m) 19s ago 4m 76.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:54:07.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4m) 10s ago 4m 77.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:54:07.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4m) 19s ago 4m 81.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:54:07.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4m) 10s ago 4m 79.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: mgrmap e27: x(active, since 41s), standbys: y 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:07 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:07.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return _run_code(code, main_globals, None, 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: exec(code, run_globals) 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:07.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.wait_async( 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return future.result(timeout) 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return self.__get_result() 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: raise self._exception 2024-09-20T19:54:07.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: out, err, code = await self._run_cephadm( 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: raise OrchestratorError( 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Creating ceph-iscsi config... 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:07.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: Traceback (most recent call last): 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: return _run_code(code, main_globals, None, 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: exec(code, run_globals) 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:07.395 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:07.479 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:54:07.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:07.481 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:54:07.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T19:54:07.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:54:07.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:54:07.481 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: mgrmap e27: x(active, since 41s), standbys: y 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: mgrmap e27: x(active, since 41s), standbys: y 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:07.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:07.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:07.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:07 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [], 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "", 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T19:54:07.760 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:54:07.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state', 'daemon prometheus.a on smithi196 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state', 'daemon prometheus.a on smithi196 is in unknown state'] 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout: daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout: daemon prometheus.a on smithi196 is in unknown state 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:08.078 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.wait_async( 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.301 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.302 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.wait_async( 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: return future.result(timeout) 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.__get_result() 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise self._exception 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:08.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.c.smithi184.stdout: return future.result(timeout) 2024-09-20T19:54:08.305 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.__get_result() 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise self._exception 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/3267096946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2629677980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-20T19:54:08.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3267096946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:08.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:08 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2629677980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.wait_async( 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-09-20T19:54:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise OrchestratorError( 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-09-20T19:54:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3267096946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:08 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2629677980' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:09 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:09.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:09.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-20T19:54:09.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:09 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:11 smithi196 ceph-mon[25662]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-20T19:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:11 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:11 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[23692]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-20T19:54:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[30019]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-20T19:54:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:11 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T19:54:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:13 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:13.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:13.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:13.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:13.660 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.660 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:13.660 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:13 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/361175067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1862027246' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[23692]: Health check update: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/361175067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1862027246' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:14 smithi184 ceph-mon[30019]: Health check update: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:54:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T19:54:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/361175067' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1862027246' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]: dispatch 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-mon[25662]: Health check update: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:15.140 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:14 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:54:14] "GET /metrics HTTP/1.1" 200 38670 "" "Prometheus/2.43.0" 2024-09-20T19:54:15.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]': finished 2024-09-20T19:54:15.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:54:15.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]': finished 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:15.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:15 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: from='client.? ' entity='client.iscsi.foo.smithi184.evkrpb' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.184:0/125266441"}]': finished 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:15 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:16 smithi184 ceph-mon[23692]: Upgrade: Updating mgr.y 2024-09-20T19:54:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:16 smithi184 ceph-mon[23692]: Deploying daemon mgr.y on smithi184 2024-09-20T19:54:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:16 smithi184 ceph-mon[30019]: Upgrade: Updating mgr.y 2024-09-20T19:54:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:16 smithi184 ceph-mon[30019]: Deploying daemon mgr.y on smithi184 2024-09-20T19:54:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:16 smithi196 ceph-mon[25662]: Upgrade: Updating mgr.y 2024-09-20T19:54:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:16 smithi196 ceph-mon[25662]: Deploying daemon mgr.y on smithi184 2024-09-20T19:54:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:17 smithi196 ceph-mon[25662]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:17.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:17 smithi184 ceph-mon[23692]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:17.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:17 smithi184 ceph-mon[30019]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:18.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:18 smithi184 systemd[1]: Stopping Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:18.621 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:18 smithi184 podman[86118]: 2024-09-20 19:54:18.34218804 +0000 UTC m=+0.141538573 container died c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T19:54:19.030 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:18 smithi184 podman[86118]: 2024-09-20 19:54:18.727493824 +0000 UTC m=+0.526844386 container cleanup c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2) 2024-09-20T19:54:19.030 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:18 smithi184 bash[86118]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y 2024-09-20T19:54:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:19 smithi196 ceph-mon[25662]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-09-20T19:54:19.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:19 smithi184 ceph-mon[23692]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-09-20T19:54:19.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:19 smithi184 ceph-mon[30019]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-09-20T19:54:20.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:19 smithi184 podman[86158]: 2024-09-20 19:54:19.773929015 +0000 UTC m=+1.429364481 container remove c77e81976234c81f0f3717566adb7208c24f9037de01c38b5e452cd5a9360e6f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T19:54:20.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:19 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:54:20.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:20 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T19:54:20.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:20 smithi184 systemd[1]: Stopped Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:20.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:20 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.y.service: Consumed 6.295s CPU time. 2024-09-20T19:54:20.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:20 smithi184 systemd[1]: Starting Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:21.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:21 smithi184 ceph-mon[23692]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 442 B/s rd, 0 op/s 2024-09-20T19:54:21.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:21 smithi184 ceph-mon[30019]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 442 B/s rd, 0 op/s 2024-09-20T19:54:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:21 smithi196 ceph-mon[25662]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 442 B/s rd, 0 op/s 2024-09-20T19:54:21.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 podman[86500]: 2024-09-20 19:54:21.482766787 +0000 UTC m=+0.016875875 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:54:21.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 podman[86500]: 2024-09-20 19:54:21.589953255 +0000 UTC m=+0.124062343 container create 478ddee08a13978106e05fdc75e60122c6502bbfb4d35ac89f520a44f5ba6d30 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T19:54:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 podman[86500]: 2024-09-20 19:54:21.834440422 +0000 UTC m=+0.368549516 container init 478ddee08a13978106e05fdc75e60122c6502bbfb4d35ac89f520a44f5ba6d30 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T19:54:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 podman[86500]: 2024-09-20 19:54:21.837406931 +0000 UTC m=+0.371516026 container start 478ddee08a13978106e05fdc75e60122c6502bbfb4d35ac89f520a44f5ba6d30 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:54:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 ceph-mgr[86531]: -- 172.21.15.184:0/681445257 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc287890e0 con 0x55dc28767400 2024-09-20T19:54:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 bash[86500]: 478ddee08a13978106e05fdc75e60122c6502bbfb4d35ac89f520a44f5ba6d30 2024-09-20T19:54:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:21 smithi184 systemd[1]: Started Ceph mgr.y for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:22.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:22 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:22.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:22.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:22.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:22 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:23.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:23.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:23 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:23.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:23 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.732 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.732 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.732 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:24.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:24.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:24 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:54:24] "GET /metrics HTTP/1.1" 200 38670 "" "Prometheus/2.43.0" 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:25 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:25.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:25 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:26 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:26.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:26 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.035+0000 7f1cbce21040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.035+0000 7f1cbce21040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:54:27.227 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ^ 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: SyntaxError: EOL while scanning string literal 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.036+0000 7f1cbce21040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.036+0000 7f1cbce21040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:54:27.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.225+0000 7f1cbce21040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:27] ENGINE Bus STARTING 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: CherryPy Checker: 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: The Application mounted at '' has an empty config. 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:27] ENGINE Serving on http://:::9283 2024-09-20T19:54:27.525 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:27] ENGINE Bus STARTED 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Standby manager daemon y restarted 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: Standby manager daemon y started 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Standby manager daemon y restarted 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: Standby manager daemon y started 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:27 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Standby manager daemon y restarted 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: Standby manager daemon y started 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:27 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:28 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:28.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:28.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:28.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:28.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:28 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.779 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.782 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:29 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:29.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:29.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:29.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:29 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.914 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.916 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:30 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:30.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:30.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:30.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:30.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:30.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:30 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:31 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:31.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:31 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:32 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:32.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:32 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:33.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:33.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:33 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:33.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:33 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:34 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:34.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:34 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:54:34] "GET /metrics HTTP/1.1" 200 38666 "" "Prometheus/2.43.0" 2024-09-20T19:54:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:35 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:35.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:35.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:35.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:35 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.825 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.828 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.835 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.836 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.837 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.838 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.838 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:36 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:36.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:36 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:37.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:37 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:37 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.352 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:38] ENGINE Bus STOPPING 2024-09-20T19:54:38.555 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:38.161+0000 7f9796699640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T19:54:38.555 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ignoring --setuser ceph since I am not root 2024-09-20T19:54:38.556 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ignoring --setgroup ceph since I am not root 2024-09-20T19:54:38.556 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mgr[56169]: -- 172.21.15.196:0/409245617 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c243330e0 con 0x560c24311000 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Activating manager daemon y 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:54:38.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.24928 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: mgrmap e29: y(active, starting, since 0.00829077s) 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Activating manager daemon y 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.24928 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: mgrmap e29: y(active, starting, since 0.00829077s) 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:54:38.649 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:54:38.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[30019]: Manager daemon y is now available 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:54:38.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-mon[23692]: Manager daemon y is now available 2024-09-20T19:54:38.652 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:54:38.652 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:38] ENGINE Bus STOPPED 2024-09-20T19:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:38.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 172.21.15.196:0/2485296868' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Activating manager daemon y 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.24928 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: mgrmap e29: y(active, starting, since 0.00829077s) 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:54:38.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:54:38.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:38 smithi196 ceph-mon[25662]: Manager daemon y is now available 2024-09-20T19:54:39.196 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:39] ENGINE Bus STARTING 2024-09-20T19:54:39.196 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:54:39.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:39] ENGINE Serving on http://:::9283 2024-09-20T19:54:39.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:54:39] ENGINE Bus STARTED 2024-09-20T19:54:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:54:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: mgrmap e30: y(active, since 1.00913s) 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: mgrmap e30: y(active, since 1.00913s) 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:39 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 11m - - 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 11m - - 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5m) 1s ago 6m 27.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6m) 12s ago 6m 89.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (27s) 1s ago 4m 54.3M - 3.8 4404209dfb71 954db17dd2c1 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,9283,8765 running (80s) 12s ago 10m 501M - 19.3.0-5127-ge8baca1e 82f5f45939a4 0f49b238ddd4 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (18s) 1s ago 12m 215M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:54:40.058 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (12m) 1s ago 12m 60.1M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (10m) 12s ago 10m 52.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (10m) 1s ago 10m 48.3M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6m) 1s ago 6m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6m) 12s ago 6m 16.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (10m) 1s ago 10m 58.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (9m) 1s ago 9m 62.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (9m) 1s ago 9m 56.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (9m) 1s ago 9m 50.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (8m) 12s ago 8m 63.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (8m) 12s ago 8m 52.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7m) 12s ago 7m 49.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7m) 12s ago 7m 52.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (46s) 12s ago 7m 37.6M - 2.43.0 a07b618ecd1d 2192425c9ef6 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5m) 1s ago 5m 77.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5m) 12s ago 5m 78.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:54:40.060 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4m) 1s ago 4m 81.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:54:40.060 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4m) 12s ago 4m 79.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: mgrmap e30: y(active, since 1.00913s) 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-09-20T19:54:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:39 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:40.104+0000 7f1c1f2f4640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:54:40] ENGINE Error in HTTPServer.serve 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._connections.run(self.expiration_interval) 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._run(expiration_interval) 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:54:40.322 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: s = self.context.wrap_socket( 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.sslsocket_class._create( 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self.do_handshake() 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._sslobj.do_handshake() 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:54:40.323 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:54:40.444 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:54:40.445 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T19:54:40.445 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:54:40.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:54:40.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:54:40.770 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:54:40.770 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:54:40.770 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr" 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "", 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:54:40.771 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: mgrmap e31: y(active, since 2s) 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:40.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:40.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1101899137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:40.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:40 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: mgrmap e31: y(active, since 2s) 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1101899137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: mgrmap e31: y(active, since 2s) 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1101899137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:54:41.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:40 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.090 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:41.090 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: [20/Sep/2024:19:54:39] ENGINE Bus STARTING 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: [20/Sep/2024:19:54:40] ENGINE Serving on https://172.21.15.184:7150 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: [20/Sep/2024:19:54:40] ENGINE Error in HTTPServer.serve 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:54:41.892 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: [20/Sep/2024:19:54:40] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: [20/Sep/2024:19:54:40] ENGINE Bus STARTED 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: [20/Sep/2024:19:54:39] ENGINE Bus STARTING 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: [20/Sep/2024:19:54:40] ENGINE Serving on https://172.21.15.184:7150 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: [20/Sep/2024:19:54:40] ENGINE Error in HTTPServer.serve 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:54:41.894 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.do_handshake() 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:54:41.895 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: [20/Sep/2024:19:54:40] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: [20/Sep/2024:19:54:40] ENGINE Bus STARTED 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2772935637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:41.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:41.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2772935637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:41.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:41 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: [20/Sep/2024:19:54:39] ENGINE Bus STARTING 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: [20/Sep/2024:19:54:40] ENGINE Serving on https://172.21.15.184:7150 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: [20/Sep/2024:19:54:40] ENGINE Error in HTTPServer.serve 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:41.957 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:54:41.958 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: [20/Sep/2024:19:54:40] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: [20/Sep/2024:19:54:40] ENGINE Bus STARTED 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2772935637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:54:41.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:41 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: mgrmap e32: y(active, since 4s) 2024-09-20T19:54:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: mgrmap e32: y(active, since 4s) 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:42 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: mgrmap e32: y(active, since 4s) 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:42 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:43.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.299+0000 7f7e295aa040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:54:43.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.299+0000 7f7e295aa040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ^ 2024-09-20T19:54:43.391 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: SyntaxError: EOL while scanning string literal 2024-09-20T19:54:43.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.300+0000 7f7e295aa040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:54:43.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.300+0000 7f7e295aa040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:54:43.635 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:43.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:43.636 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:43.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.487+0000 7f7e295aa040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:54:43] ENGINE Bus STARTING 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: CherryPy Checker: 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: The Application mounted at '' has an empty config. 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:54:43] ENGINE Serving on http://:::9283 2024-09-20T19:54:43.743 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: [20/Sep/2024:19:54:43] ENGINE Bus STARTED 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[23692]: Standby manager daemon x started 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:43 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:43 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[56165]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:54:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-mon[25662]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-mon[25662]: mgrmap e33: y(active, since 6s), standbys: x 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:44 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:45.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[23692]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[23692]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[23692]: mgrmap e33: y(active, since 6s), standbys: x 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[30019]: from='client.25072 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[30019]: mgrmap e33: y(active, since 6s), standbys: x 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T19:54:45.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:44 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:45 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:45 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[23692]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:54:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[23692]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[30019]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:54:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:46 smithi184 ceph-mon[30019]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:46 smithi196 ceph-mon[25662]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:54:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:46 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:46 smithi196 ceph-mon[25662]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:54:47.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:54:47.660+0000 7f1c4c34e640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:47.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Creating ceph-iscsi config... 2024-09-20T19:54:47.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:47.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:47.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return _run_code(code, main_globals, None, 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: exec(code, run_globals) 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:47.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:47.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.wait_async( 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return future.result(timeout) 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.__get_result() 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: raise self._exception 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: out, err, code = await self._run_cephadm( 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: raise OrchestratorError( 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:47.802 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Creating ceph-iscsi config... 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return _run_code(code, main_globals, None, 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: exec(code, run_globals) 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:47.803 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:47.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:47.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:47.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:47.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:47.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:54:47 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:48 smithi196 ceph-mon[25662]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.wait_async( 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-09-20T19:54:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise OrchestratorError( 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:48 smithi196 ceph-mon[25662]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:54:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:48 smithi196 ceph-mon[25662]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[23692]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.wait_async( 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: return future.result(timeout) 2024-09-20T19:54:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[30019]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:54:49.302 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.wait_async( 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: return future.result(timeout) 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.__get_result() 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise self._exception 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.304 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[23692]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[23692]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.__get_result() 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise self._exception 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:54:49.306 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:54:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:54:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[30019]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T19:54:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:48 smithi184 ceph-mon[30019]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[23692]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:54:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[30019]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:54:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:49 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:49 smithi196 ceph-mon[25662]: Reconfiguring daemon prometheus.a on smithi196 2024-09-20T19:54:50.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:49 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:49 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:49 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:50.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:50 smithi184 ceph-mon[23692]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:54:50.892 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:50 smithi184 ceph-mon[23692]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:50.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:50 smithi184 ceph-mon[30019]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:54:50.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:50 smithi184 ceph-mon[30019]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:50.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:50 smithi196 ceph-mon[25662]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:54:50.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:50 smithi196 ceph-mon[25662]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:51.342 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 systemd[1]: Stopping Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:51.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.340Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T19:54:51.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T19:54:51.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T19:54:51.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T19:54:51.640 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[59462]: ts=2024-09-20T19:54:51.341Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T19:54:51.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 podman[64996]: 2024-09-20 19:54:51.351051563 +0000 UTC m=+0.122101410 container died 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.006 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 podman[64996]: 2024-09-20 19:54:51.639917455 +0000 UTC m=+0.410967309 container cleanup 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.007 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 bash[64996]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T19:54:52.007 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:51 smithi196 podman[65008]: 2024-09-20 19:54:51.874477059 +0000 UTC m=+0.522849927 container remove 2192425c9ef62ab804de2ca440ad3f30d43da2149709d2bb5f2d70fa7425b763 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 systemd[1]: Stopped Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.092s CPU time. 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 systemd[1]: Starting Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 podman[65096]: 2024-09-20 19:54:52.132626204 +0000 UTC m=+0.016450456 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T19:54:52.360 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 podman[65096]: 2024-09-20 19:54:52.25887847 +0000 UTC m=+0.142702722 container create 9515b5acbfddfa6b56b768112ed718480c152a7fc6a4cf3e1bf3d4b4ab008afc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 podman[65096]: 2024-09-20 19:54:52.461231707 +0000 UTC m=+0.345055964 container init 9515b5acbfddfa6b56b768112ed718480c152a7fc6a4cf3e1bf3d4b4ab008afc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 podman[65096]: 2024-09-20 19:54:52.463833904 +0000 UTC m=+0.347658156 container start 9515b5acbfddfa6b56b768112ed718480c152a7fc6a4cf3e1bf3d4b4ab008afc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.487Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.487Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.487Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi196 (none))" 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.487Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.487Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T19:54:52.856 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.489Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.489Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.490Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.490Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.492Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.492Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.029µs 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.492Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.493Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.503Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.504Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.504Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.504Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.732µs wal_replay_duration=11.878122ms wbl_replay_duration=429ns total_replay_duration=11.932574ms 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.506Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T19:54:52.857 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.506Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.506Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.525Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.224455ms db_storage=1.075µs remote_storage=1.212µs web_handler=365ns query_engine=551ns scrape=413.688µs scrape_sd=119.408µs notify=17.05µs notify_sd=24.231µs rules=18.309536ms tracing=5.431µs 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.525Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T19:54:52.525Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 bash[65096]: 9515b5acbfddfa6b56b768112ed718480c152a7fc6a4cf3e1bf3d4b4ab008afc 2024-09-20T19:54:52.858 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 19:54:52 smithi196 systemd[1]: Started Ceph prometheus.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:52 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:52 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:53.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:54:53.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:54:54.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:53 smithi196 ceph-mon[25662]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:54:54.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:53 smithi196 ceph-mon[25662]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:54.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:53 smithi196 ceph-mon[25662]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:54:54.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:53 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:54.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:53 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[23692]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:54:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[23692]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[23692]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[30019]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[30019]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[30019]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:53 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:54 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:54 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:54 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:54:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:54:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:54 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:56.047 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:55 smithi196 systemd[1]: Stopping Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:56.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:55 smithi196 ceph-mon[25662]: Upgrade: Updating mgr.x 2024-09-20T19:54:56.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:55 smithi196 ceph-mon[25662]: Deploying daemon mgr.x on smithi196 2024-09-20T19:54:56.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:55 smithi196 ceph-mon[25662]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[23692]: Upgrade: Updating mgr.x 2024-09-20T19:54:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[23692]: Deploying daemon mgr.x on smithi196 2024-09-20T19:54:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[23692]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[30019]: Upgrade: Updating mgr.x 2024-09-20T19:54:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[30019]: Deploying daemon mgr.x on smithi196 2024-09-20T19:54:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:55 smithi184 ceph-mon[30019]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 485 B/s rd, 0 op/s 2024-09-20T19:54:56.301 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 podman[65366]: 2024-09-20 19:54:56.045955835 +0000 UTC m=+0.122437058 container died 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T19:54:56.597 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 podman[65366]: 2024-09-20 19:54:56.374262747 +0000 UTC m=+0.450743969 container cleanup 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:54:56.598 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 bash[65366]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x 2024-09-20T19:54:56.848 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 podman[65378]: 2024-09-20 19:54:56.596713018 +0000 UTC m=+0.550902094 container remove 0f49b238ddd41d55fa4d91cedd4448ca41671f13a9c7127c67c7dba2e72a31fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:54:56.848 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:54:56.848 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T19:54:56.848 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 systemd[1]: Stopped Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:56.848 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:56 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Consumed 46.396s CPU time. 2024-09-20T19:54:57.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 systemd[1]: Starting Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:54:57.510 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 podman[65509]: 2024-09-20 19:54:57.159912266 +0000 UTC m=+0.015922497 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:54:57.510 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 podman[65509]: 2024-09-20 19:54:57.287919271 +0000 UTC m=+0.143929488 container create dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T19:54:57.510 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 podman[65509]: 2024-09-20 19:54:57.509413967 +0000 UTC m=+0.365424188 container init dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T19:54:57.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 podman[65509]: 2024-09-20 19:54:57.512442845 +0000 UTC m=+0.368453069 container start dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:54:57.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mgr[65528]: -- 172.21.15.196:0/3163571843 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592dd3190e0 con 0x5592dd2f6800 2024-09-20T19:54:57.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 bash[65509]: dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c 2024-09-20T19:54:57.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:54:57 smithi196 systemd[1]: Started Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:58.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:58.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:58.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:58.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:58.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:57 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:57 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:54:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:58 smithi196 ceph-mon[25662]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:58 smithi196 ceph-mon[25662]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:58 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:58 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:58 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:59.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[23692]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[23692]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:59.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:59.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:54:59.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[30019]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[30019]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:54:59.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:54:59.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:54:59.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:58 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:55:00.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:54:59 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/2054214478' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:00.099 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:54:59 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2054214478' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:54:59 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2054214478' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:00 smithi184 ceph-mon[23692]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 135 B/s rd, 0 op/s 2024-09-20T19:55:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:00 smithi184 ceph-mon[30019]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 135 B/s rd, 0 op/s 2024-09-20T19:55:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:00 smithi196 ceph-mon[25662]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 135 B/s rd, 0 op/s 2024-09-20T19:55:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:01 smithi196 ceph-mon[25662]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-20T19:55:02.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:01 smithi184 ceph-mon[23692]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-20T19:55:02.166 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:01 smithi184 ceph-mon[30019]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-20T19:55:02.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:55:02] "GET /metrics HTTP/1.1" 200 38664 "" "Prometheus/2.43.0" 2024-09-20T19:55:02.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:02.220+0000 7f1c4c34e640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Creating ceph-iscsi config... 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:02.549 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return _run_code(code, main_globals, None, 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: exec(code, run_globals) 2024-09-20T19:55:02.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:02.551 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:02.551 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:02.551 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:02.551 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:02.551 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:55:02.552 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.wait_async( 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return future.result(timeout) 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.__get_result() 2024-09-20T19:55:02.553 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: raise self._exception 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: out, err, code = await self._run_cephadm( 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: raise OrchestratorError( 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:02.554 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Creating ceph-iscsi config... 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:02.555 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return _run_code(code, main_globals, None, 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: exec(code, run_globals) 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:02.556 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:02.557 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:02.620 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.319+0000 7f4e8fc89040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.319+0000 7f4e8fc89040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: ^ 2024-09-20T19:55:02.621 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: SyntaxError: EOL while scanning string literal 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.320+0000 7f4e8fc89040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.320+0000 7f4e8fc89040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.512+0000 7f4e8fc89040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: [20/Sep/2024:19:55:02] ENGINE Bus STARTING 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: CherryPy Checker: 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: The Application mounted at '' has an empty config. 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: [20/Sep/2024:19:55:02] ENGINE Serving on http://:::9283 2024-09-20T19:55:02.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[65524]: [20/Sep/2024:19:55:02] ENGINE Bus STARTED 2024-09-20T19:55:03.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.021 INFO:journalctl@ceph.mon.a.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.021 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.021 INFO:journalctl@ceph.mon.a.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.022 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.023 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.wait_async( 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: return future.result(timeout) 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.__get_result() 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:55:03.024 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise self._exception 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.025 INFO:journalctl@ceph.mon.a.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.027 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.wait_async( 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: return future.result(timeout) 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.__get_result() 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise self._exception 2024-09-20T19:55:03.029 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: raise OrchestratorError( 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.030 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: exec(code, run_globals) 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.031 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 168 B/s rd, 0 op/s 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Standby manager daemon x restarted 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[23692]: Standby manager daemon x started 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 168 B/s rd, 0 op/s 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Standby manager daemon x restarted 2024-09-20T19:55:03.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:02 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.wait_async( 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-09-20T19:55:03.142 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: out, err, code = await self._run_cephadm( 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise OrchestratorError( 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi-foo-smithi184-evkrpb 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: /usr/bin/podman: stderr Error: no such container ceph-4b598dd2-7788-11ef-baf5-efdc52797490-iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Reconfig daemon iscsi.foo.smithi184.evkrpb ... 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/iscsi-gateway.cfg 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/iscsi.foo.smithi184.evkrpb/tcmu-runner-entrypoint.sh 2024-09-20T19:55:03.143 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T19:55:03.144 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb: Job for ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service failed because the control process exited with error code. 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" and "journalctl -xeu ceph-4b598dd2-7788-11ef-baf5-efdc52797490@iscsi.foo.smithi184.evkrpb.service" for details. 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 168 B/s rd, 0 op/s 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Reconfiguring iscsi.foo.smithi184.evkrpb (dependencies changed)... 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi184.evkrpb", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Reconfiguring daemon iscsi.foo.smithi184.evkrpb on smithi184 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Standby manager daemon x restarted 2024-09-20T19:55:03.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:02 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:55:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:03.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:03.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:04 smithi184 ceph-mon[23692]: mgrmap e34: y(active, since 25s), standbys: x 2024-09-20T19:55:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:04 smithi184 ceph-mon[30019]: mgrmap e34: y(active, since 25s), standbys: x 2024-09-20T19:55:04.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:04 smithi196 ceph-mon[25662]: mgrmap e34: y(active, since 25s), standbys: x 2024-09-20T19:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T19:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:05 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:05.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:05 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/4217001056' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/4217001056' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:06 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/4217001056' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:06.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:06 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi184.evkrpb on smithi184 is in unknown state']) 2024-09-20T19:55:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:07 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:07 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:07.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:07 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:08 smithi184 ceph-mon[23692]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:08 smithi184 ceph-mon[30019]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:08.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:08 smithi196 ceph-mon[25662]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:55:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:09 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:10.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:09 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:10.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:09 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:10.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:09 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:55:10.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:09 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:55:10.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:09 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:10.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:10.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: Upgrade: Updating mgr.x 2024-09-20T19:55:10.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: Deploying daemon mgr.x on smithi196 2024-09-20T19:55:10.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:10.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:10.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:10 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: Upgrade: Updating mgr.x 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: Deploying daemon mgr.x on smithi196 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-09-20T19:55:11.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: Upgrade: Updating mgr.x 2024-09-20T19:55:11.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: Deploying daemon mgr.x on smithi196 2024-09-20T19:55:11.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:10 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 11m - - 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 11m - - 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6m) 2s ago 7m 27.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7m) 6s ago 6m 90.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7s) 2s ago 5m 71.1M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:8443,9283,8765 running (112s) 6s ago 11m 351M - 19.3.0-5127-ge8baca1e 82f5f45939a4 0f49b238ddd4 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (50s) 2s ago 12m 496M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (12m) 2s ago 12m 61.9M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (11m) 6s ago 11m 50.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (11m) 2s ago 11m 46.7M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:55:11.960 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6m) 2s ago 6m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6m) 6s ago 6m 16.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (10m) 2s ago 10m 59.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (10m) 2s ago 10m 62.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (9m) 2s ago 9m 56.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (9m) 2s ago 9m 51.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (9m) 6s ago 9m 64.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (8m) 6s ago 8m 53.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (8m) 6s ago 8m 50.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (8m) 6s ago 8m 52.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (78s) 6s ago 7m 39.9M - 2.43.0 a07b618ecd1d 2192425c9ef6 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5m) 2s ago 5m 78.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5m) 6s ago 5m 79.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5m) 2s ago 5m 82.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:55:11.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5m) 6s ago 5m 80.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:55:12.240 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:55:12] "GET /metrics HTTP/1.1" 200 38664 "" "Prometheus/2.43.0" 2024-09-20T19:55:12.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:55:12.283 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr" 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:55:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T19:55:12.552 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T19:55:12.552 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:55:12.552 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:55:12.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:12 smithi196 ceph-mon[25662]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 1 op/s 2024-09-20T19:55:12.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:12 smithi196 ceph-mon[25662]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:12 smithi196 ceph-mon[25662]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:12 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1621940491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:12.860 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:12.860 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:12.860 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:12.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[23692]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 1 op/s 2024-09-20T19:55:12.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[23692]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[23692]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/1621940491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:12.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[30019]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 1 op/s 2024-09-20T19:55:12.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[30019]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[30019]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:12.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:12 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/1621940491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:13.389 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:13 smithi196 systemd[1]: Stopping Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:55:13.718 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:13 smithi196 podman[66931]: 2024-09-20 19:55:13.428294177 +0000 UTC m=+0.131871580 container died dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T19:55:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:13.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:13.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/642778597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='client.? 172.21.15.184:0/642778597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:13 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:13 smithi196 podman[66931]: 2024-09-20 19:55:13.803770036 +0000 UTC m=+0.507347428 container cleanup dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:13 smithi196 bash[66931]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/642778597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.057 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.057 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:13 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:14.320 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 podman[66974]: 2024-09-20 19:55:14.05484738 +0000 UTC m=+0.626322550 container remove dfe253bf275148a0f175440a175edaf75dbbe5cf8d9ca51c3dc91c4aff676a0c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T19:55:14.320 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T19:55:14.320 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T19:55:14.320 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: Stopped Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:55:14.320 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mgr.x.service: Consumed 5.860s CPU time. 2024-09-20T19:55:14.631 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: Starting Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:14 smithi196 ceph-mon[25662]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:14 smithi196 ceph-mon[25662]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:14 smithi196 ceph-mon[25662]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 podman[67103]: 2024-09-20 19:55:14.533320236 +0000 UTC m=+0.016251298 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 podman[67103]: 2024-09-20 19:55:14.673101227 +0000 UTC m=+0.156032323 container create ad8e16e70e793a0c88f6e3a3f86bef32a9e2040da7ab415ab48b59815f504f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 podman[67103]: 2024-09-20 19:55:14.875526585 +0000 UTC m=+0.358457651 container init ad8e16e70e793a0c88f6e3a3f86bef32a9e2040da7ab415ab48b59815f504f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T19:55:14.890 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 podman[67103]: 2024-09-20 19:55:14.878582667 +0000 UTC m=+0.361513727 container start ad8e16e70e793a0c88f6e3a3f86bef32a9e2040da7ab415ab48b59815f504f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:55:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[23692]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-09-20T19:55:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[23692]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T19:55:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[23692]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:55:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[30019]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-09-20T19:55:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[30019]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T19:55:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:14 smithi184 ceph-mon[30019]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:55:15.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 ceph-mgr[67122]: -- 172.21.15.196:0/2039588855 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e9453b0e0 con 0x564e94519400 2024-09-20T19:55:15.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 bash[67103]: ad8e16e70e793a0c88f6e3a3f86bef32a9e2040da7ab415ab48b59815f504f9b 2024-09-20T19:55:15.390 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:14 smithi196 systemd[1]: Started Ceph mgr.x for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:55:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:16 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:16.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:16.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:16.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:16 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state']) 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.175 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:17 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state']) 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.x on smithi196 is in unknown state']) 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:17.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:17.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:17.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:17 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:18 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:18.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:18 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:19 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:19.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:19.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:19 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.105+0000 7ff4df4db040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.105+0000 7ff4df4db040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:55:20.392 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: ^ 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: SyntaxError: EOL while scanning string literal 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.106+0000 7ff4df4db040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.106+0000 7ff4df4db040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.291+0000 7ff4df4db040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:20.393 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:20] ENGINE Bus STARTING 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: CherryPy Checker: 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: The Application mounted at '' has an empty config. 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1 op/s 2024-09-20T19:55:20.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:20.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:20 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:20.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:20] ENGINE Serving on http://:::9283 2024-09-20T19:55:20.710 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:20] ENGINE Bus STARTED 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Standby manager daemon x restarted 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:21 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Standby manager daemon x restarted 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Standby manager daemon x started 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Standby manager daemon x restarted 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:21.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:21 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: mgrmap e35: y(active, since 43s), standbys: x 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:22 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:55:22] "GET /metrics HTTP/1.1" 200 38664 "" "Prometheus/2.43.0" 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: mgrmap e35: y(active, since 43s), standbys: x 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: mgrmap e35: y(active, since 43s), standbys: x 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:22.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:22 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:23 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:23.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:23.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:23.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:24 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:24.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:24 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:25 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:25.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:25 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:26 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:26.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:26 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.153 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.153 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.156 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.157 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.158 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.159 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.159 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.159 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:27 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:27 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:28 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:28.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:28.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:28 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Upgrade: Setting container_image for all mgr 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T19:55:29.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T19:55:29.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T19:55:29.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T19:55:29.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:55:29.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T19:55:29.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:29 smithi196 ceph-mon[25662]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:29.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:29.392 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Upgrade: Setting container_image for all mgr 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Upgrade: Setting container_image for all mgr 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T19:55:29.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[30019]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T19:55:29.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:29 smithi184 ceph-mon[23692]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:30.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:30 smithi196 ceph-mon[25662]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:30.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:30 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:30 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:30 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[23692]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[30019]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:30.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:30.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:30 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:32.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:55:32] "GET /metrics HTTP/1.1" 200 38658 "" "Prometheus/2.43.0" 2024-09-20T19:55:32.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: Upgrade: Updating mon.a 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:32.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:32 smithi196 ceph-mon[25662]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: Upgrade: Updating mon.a 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[23692]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: Upgrade: Updating mon.a 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:55:33.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:32 smithi184 ceph-mon[30019]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:33.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:33 smithi184 systemd[1]: Stopping Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:55:33.659 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:33.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:33.659 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:33.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:34.011 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:33] ENGINE Bus STOPPING 2024-09-20T19:55:34.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[23688]: 2024-09-20T19:55:33.657+0000 7f273e534640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:55:34.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[23688]: 2024-09-20T19:55:33.657+0000 7f273e534640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T19:55:34.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:33 smithi184 podman[97649]: 2024-09-20 19:55:33.780079309 +0000 UTC m=+0.242978746 container died 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T19:55:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 podman[97649]: 2024-09-20 19:55:34.174078796 +0000 UTC m=+0.636978229 container cleanup 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS) 2024-09-20T19:55:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 bash[97649]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:55:34.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:55:34.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:34] ENGINE Bus STOPPED 2024-09-20T19:55:34.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:34] ENGINE Bus STARTING 2024-09-20T19:55:34.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:34] ENGINE Serving on http://:::9283 2024-09-20T19:55:34.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:34] ENGINE Bus STARTED 2024-09-20T19:55:34.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 podman[97661]: 2024-09-20 19:55:34.417284673 +0000 UTC m=+0.635959221 container remove 0a53f2d7068358a3903086c7b45a885fe104b5c57fd16913fd3b9e87a8a57260 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:55:34.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T19:55:34.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 systemd[1]: Stopped Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:55:34.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Consumed 17.895s CPU time. 2024-09-20T19:55:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 systemd[1]: Starting Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:55:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 podman[97802]: 2024-09-20 19:55:34.976299678 +0000 UTC m=+0.113309429 container create 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T19:55:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:34 smithi184 podman[97802]: 2024-09-20 19:55:34.879440881 +0000 UTC m=+0.016450602 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:55:35.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 podman[97802]: 2024-09-20 19:55:35.170101028 +0000 UTC m=+0.307110746 container init 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:55:35.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 podman[97802]: 2024-09-20 19:55:35.173289286 +0000 UTC m=+0.310299003 container start 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: pidfile_write: ignore empty --pid-file 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: load: jerasure load: lrc 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Git sha 0 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: DB SUMMARY 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: DB Session ID: ZJHRQCTBQ3NMFNEAFW29 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: MANIFEST file: MANIFEST-000015 size: 2490 Bytes 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000044.sst 2024-09-20T19:55:35.331 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000042.log size: 491614 ; 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.env: 0x557addcf2120 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.info_log: 0x557ade567700 2024-09-20T19:55:35.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.statistics: (nil) 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.use_fsync: 0 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:55:35.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.db_log_dir: 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.wal_dir: 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:55:35.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.write_buffer_manager: 0x557ade56a1e0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.unordered_write: 0 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:55:35.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.row_cache: None 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.wal_filter: None 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.wal_compression: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:55:35.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:55:35.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_open_files: -1 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Compression algorithms supported: 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kZSTD supported: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:55:35.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.merge_operator: 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_filter: None 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:55:35.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557ade567320) 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_type: 0 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_index_type: 0 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_shortening: 1 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: checksum: 4 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: no_block_cache: 0 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache: 0x557ade55b610 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_options: 2024-09-20T19:55:35.340 INFO:journalctl@ceph.mon.a.smithi184.stdout: capacity : 536870912 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_shard_bits : 4 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: strict_capacity_limit : 0 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_compressed: (nil) 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: persistent_cache: (nil) 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size: 4096 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size_deviation: 10 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_restart_interval: 16 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_block_restart_interval: 1 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: metadata_block_size: 4096 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: partition_filters: 0 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: use_delta_encoding: 1 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: filter_policy: bloomfilter 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: whole_key_filtering: 1 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: verify_compression: 0 2024-09-20T19:55:35.341 INFO:journalctl@ceph.mon.a.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: format_version: 5 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_index_compression: 1 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_align: 0 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: prepopulate_block_cache: 0 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression: NoCompression 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:55:35.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.num_levels: 7 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:55:35.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:55:35.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:55:35.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:55:35.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:55:35.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:55:35.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:55:35.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.ttl: 2592000 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:55:35.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 46, last_sequence is 21450, log_number is 42,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 42 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 42 2024-09-20T19:55:35.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862135200194, "job": 1, "event": "recovery_started", "wal_files": [42]} 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #42 mode 2 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862135202425, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 47, "file_size": 468999, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 21451, "largest_seqno": 21911, "table_properties": {"data_size": 466112, "index_size": 1079, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 7747, "raw_average_key_size": 26, "raw_value_size": 460678, "raw_average_value_size": 1556, "num_data_blocks": 44, "num_entries": 296, "num_filter_entries": 296, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862135, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3", "db_session_id": "ZJHRQCTBQ3NMFNEAFW29", "orig_file_number": 47, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862135202520, "job": 1, "event": "recovery_finished"} 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:5047] Creating manifest 49 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000042.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557ade58ae00 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: DB pointer 0x557ade68c000 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** DB Stats ** 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:55:35.350 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: L0 1/0 458.01 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 325.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: L6 1/0 8.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Sum 2/0 9.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 325.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 325.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:55:35.351 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 325.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative compaction: 0.00 GB write, 101.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval compaction: 0.00 GB write, 101.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache BinnedLRUCache@0x557ade55b610#2 capacity: 512.00 MB usage: 2.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: starting mon.a rank 0 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???) e3 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).mds e1 new map 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).mds e1 print_map 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: e1 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: legacy client fscid: -1 2024-09-20T19:55:35.353 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout: No filesystems configured 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 ceph-mon[97821]: mon.a@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 bash[97802]: 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d 2024-09-20T19:55:35.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:35 smithi184 systemd[1]: Started Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:55:38.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ignoring --setuser ceph since I am not root 2024-09-20T19:55:38.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ignoring --setgroup ceph since I am not root 2024-09-20T19:55:38.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mgr[86531]: -- 172.21.15.184:0/3822329148 <== mon.2 v2:172.21.15.184:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559e16e150e0 con 0x559e16df2800 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: mon.a calling monitor election 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: monmap epoch 3 2024-09-20T19:55:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: min_mon_release 18 (reef) 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: election_strategy: 1 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: fsmap 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: mgrmap e35: y(active, since 60s), standbys: x 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: from='mgr.25325 ' entity='' 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[97821]: mgrmap e36: y(active, since 60s), standbys: x 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: mon.a calling monitor election 2024-09-20T19:55:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: monmap epoch 3 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: min_mon_release 18 (reef) 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: election_strategy: 1 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: fsmap 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: mgrmap e35: y(active, since 60s), standbys: x 2024-09-20T19:55:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: from='mgr.25325 ' entity='' 2024-09-20T19:55:38.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:38 smithi184 ceph-mon[30019]: mgrmap e36: y(active, since 60s), standbys: x 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: ignoring --setuser ceph since I am not root 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: ignoring --setgroup ceph since I am not root 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mgr[67122]: -- 172.21.15.196:0/3436342395 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c7aba10e0 con 0x560c7ab7f000 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mgr.25325 172.21.15.184:0/1373707478' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: mon.a calling monitor election 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: monmap epoch 3 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: min_mon_release 18 (reef) 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: election_strategy: 1 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:55:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: fsmap 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: mgrmap e35: y(active, since 60s), standbys: x 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: from='mgr.25325 ' entity='' 2024-09-20T19:55:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:38 smithi196 ceph-mon[25662]: mgrmap e36: y(active, since 60s), standbys: x 2024-09-20T19:55:43.356 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-mon[25662]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:43.356 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-mon[25662]: Standby manager daemon x restarted 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-mon[25662]: Standby manager daemon x started 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.071+0000 7efc65b64040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.071+0000 7efc65b64040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:55:43.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: ^ 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: SyntaxError: EOL while scanning string literal 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.071+0000 7efc65b64040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.071+0000 7efc65b64040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.249+0000 7efc65b64040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:43] ENGINE Bus STARTING 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: CherryPy Checker: 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: The Application mounted at '' has an empty config. 2024-09-20T19:55:43.358 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: 2024-09-20T19:55:43.359 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:43] ENGINE Serving on http://:::9283 2024-09-20T19:55:43.359 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 20 19:55:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-x[67118]: [20/Sep/2024:19:55:43] ENGINE Bus STARTED 2024-09-20T19:55:43.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[97821]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[97821]: Standby manager daemon x restarted 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[97821]: Standby manager daemon x started 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[30019]: Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[30019]: Standby manager daemon x restarted 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-mon[30019]: Standby manager daemon x started 2024-09-20T19:55:43.518 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.149+0000 7fefc8ff7040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.149+0000 7fefc8ff7040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ^ 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: SyntaxError: EOL while scanning string literal 2024-09-20T19:55:43.519 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.520 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.150+0000 7fefc8ff7040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T19:55:43.520 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.150+0000 7fefc8ff7040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T19:55:43.520 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:43.336+0000 7fefc8ff7040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T19:55:43.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:43.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:43.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:43.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:44.382 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: mgrmap e37: y(active, since 65s), standbys: x 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Active manager daemon y restarted 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Activating manager daemon y 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: mgrmap e38: y(active, starting, since 0.00970486s), standbys: x 2024-09-20T19:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:55:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Manager daemon y is now available 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:55:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:44 smithi196 ceph-mon[25662]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: mgrmap e37: y(active, since 65s), standbys: x 2024-09-20T19:55:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Active manager daemon y restarted 2024-09-20T19:55:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Activating manager daemon y 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: mgrmap e38: y(active, starting, since 0.00970486s), standbys: x 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:55:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Manager daemon y is now available 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:55:44.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: mgrmap e37: y(active, since 65s), standbys: x 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Active manager daemon y restarted 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Activating manager daemon y 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: mgrmap e38: y(active, starting, since 0.00970486s), standbys: x 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T19:55:44.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Manager daemon y is now available 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.conf 2024-09-20T19:55:44.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[97821]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi184:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-mon[30019]: Updating smithi196:/var/lib/ceph/4b598dd2-7788-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:44.358+0000 7fef66d8f640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-20T19:55:44.804 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:44] ENGINE Bus STARTING 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: CherryPy Checker: 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: The Application mounted at '' has an empty config. 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:44] ENGINE Serving on http://:::9283 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:44] ENGINE Bus STARTED 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:44.519+0000 7fef2ca0d640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:19:55:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: Traceback (most recent call last): 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._connections.run(self.expiration_interval) 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._run(expiration_interval) 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:55:44.805 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: s = self.context.wrap_socket( 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self.sslsocket_class._create( 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self.do_handshake() 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self._sslobj.do_handshake() 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:55:44.806 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:55:44.950 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:55:44.950 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 12m - - 2024-09-20T19:55:44.950 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 12m - - 2024-09-20T19:55:44.950 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6m) 16s ago 7m 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7m) 15s ago 7m 92.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (40s) 16s ago 5m 54.3M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (30s) 15s ago 11m 447M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (83s) 16s ago 13m 499M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (13m) 16s ago 13m 66.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 0a53f2d70683 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (12m) 15s ago 12m 53.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (11m) 16s ago 11m 49.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7m) 16s ago 7m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7m) 15s ago 7m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (11m) 16s ago 11m 59.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (10m) 16s ago 10m 62.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:55:44.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (10m) 16s ago 10m 56.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (10m) 16s ago 10m 51.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (9m) 15s ago 9m 64.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (9m) 15s ago 9m 53.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (8m) 15s ago 8m 50.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (8m) 15s ago 8m 52.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (52s) 15s ago 8m 39.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6m) 16s ago 6m 78.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6m) 15s ago 6m 79.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5m) 16s ago 5m 82.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:55:44.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5m) 15s ago 5m 80.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:55:45.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 14, 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:55:45.274 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: mgrmap e39: y(active, since 1.01191s), standbys: x 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: [20/Sep/2024:19:55:44] ENGINE Bus STARTING 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []} : dispatch 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: [20/Sep/2024:19:55:44] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: [20/Sep/2024:19:55:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:55:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: self.do_handshake() 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: [20/Sep/2024:19:55:44] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: [20/Sep/2024:19:55:44] ENGINE Bus STARTED 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:45.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[97821]: from='client.? 172.21.15.184:0/3146587134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: mgrmap e39: y(active, since 1.01191s), standbys: x 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: [20/Sep/2024:19:55:44] ENGINE Bus STARTING 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []} : dispatch 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: [20/Sep/2024:19:55:44] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: [20/Sep/2024:19:55:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: Traceback (most recent call last): 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._run(expiration_interval) 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: s = self.context.wrap_socket( 2024-09-20T19:55:45.553 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: return self.sslsocket_class._create( 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: self.do_handshake() 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: self._sslobj.do_handshake() 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: [20/Sep/2024:19:55:44] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: [20/Sep/2024:19:55:44] ENGINE Bus STARTED 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:45.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:45 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/3146587134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:45.563 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr" 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "", 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:55:45.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: mgrmap e39: y(active, since 1.01191s), standbys: x 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: [20/Sep/2024:19:55:44] ENGINE Bus STARTING 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []} : dispatch 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: [20/Sep/2024:19:55:44] ENGINE Serving on https://172.21.15.184:7151 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: [20/Sep/2024:19:55:44] ENGINE Error in HTTPServer.serve 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._run(expiration_interval) 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:55:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: s = self.context.wrap_socket( 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.sslsocket_class._create( 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.do_handshake() 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: self._sslobj.do_handshake() 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: [20/Sep/2024:19:55:44] ENGINE Serving on http://172.21.15.184:8765 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: [20/Sep/2024:19:55:44] ENGINE Bus STARTED 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:45 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/3146587134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:45.932 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:45.932 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:45.932 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2080582276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:46.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:46 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T19:55:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='client.? 172.21.15.184:0/2080582276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='client.? 172.21.15.184:0/2080582276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi196", "caps": []} : dispatch 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:46 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:47 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:47 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: mgrmap e41: y(active, since 5s), standbys: x 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.689 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:48 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: mgrmap e41: y(active, since 5s), standbys: x 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: mgrmap e41: y(active, since 5s), standbys: x 2024-09-20T19:55:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:48.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:48.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:48.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:48.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:48 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["a"]} : dispatch 2024-09-20T19:55:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:49 smithi196 ceph-mon[25662]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:55:49.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["a"]} : dispatch 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[97821]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:55:49.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["a"]} : dispatch 2024-09-20T19:55:49.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:49 smithi184 ceph-mon[30019]: Upgrade: It appears safe to stop mon.a 2024-09-20T19:55:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:51 smithi184 ceph-mon[97821]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:51 smithi184 ceph-mon[30019]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:51.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:51 smithi196 ceph-mon[25662]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T19:55:52.115 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:55:52] "GET /metrics HTTP/1.1" 200 35860 "" "Prometheus/2.43.0" 2024-09-20T19:55:53.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:52 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:52 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:55:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:53.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:55:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:55:53.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:55:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:55:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: Upgrade: Updating mon.a 2024-09-20T19:55:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:55:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:55:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[30019]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: Upgrade: Updating mon.a 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:54 smithi184 ceph-mon[97821]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: Upgrade: Updating mon.a 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:55:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:54 smithi196 ceph-mon[25662]: Deploying daemon mon.a on smithi184 2024-09-20T19:55:55.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:55 smithi184 ceph-mon[97821]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:55.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:55 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:55 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:55:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:56 smithi184 ceph-mon[97821]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:55:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:55:56 smithi184 ceph-mon[30019]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:55:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:55:56 smithi196 ceph-mon[25662]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:55:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:57 smithi184 systemd[1]: Stopping Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:55:58.488 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE Bus STOPPING 2024-09-20T19:55:58.489 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[97817]: 2024-09-20T19:55:58.125+0000 7f02fb75f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:55:58.489 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[97817]: 2024-09-20T19:55:58.125+0000 7f02fb75f640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T19:55:58.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:55:58.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE Bus STOPPED 2024-09-20T19:55:58.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE Bus STARTING 2024-09-20T19:55:58.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE Serving on http://:::9283 2024-09-20T19:55:58.799 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:55:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:55:58] ENGINE Bus STARTED 2024-09-20T19:55:59.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:58 smithi184 podman[100223]: 2024-09-20 19:55:58.856535901 +0000 UTC m=+0.840804643 container died 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T19:55:59.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 podman[100223]: 2024-09-20 19:55:59.262170902 +0000 UTC m=+1.246439636 container cleanup 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:55:59.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 bash[100223]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a 2024-09-20T19:55:59.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 podman[100287]: 2024-09-20 19:55:59.438537632 +0000 UTC m=+1.201256818 container remove 6a5cfcb63434e23e400bf8ce0392b8f2661551410ac781d2849666d47ff6f17d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T19:56:00.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T19:56:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 systemd[1]: Stopped Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:55:59 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.a.service: Consumed 1.549s CPU time. 2024-09-20T19:56:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 systemd[1]: Starting Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:56:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 podman[100591]: 2024-09-20 19:56:00.2475753 +0000 UTC m=+0.098241539 container create 20c10837ba0d06b0d675a8b6a4c40dd47447cc49583b6aaf829230c2575a15d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-20T19:56:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 podman[100591]: 2024-09-20 19:56:00.165740333 +0000 UTC m=+0.016406574 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 podman[100591]: 2024-09-20 19:56:00.500265805 +0000 UTC m=+0.350932046 container init 20c10837ba0d06b0d675a8b6a4c40dd47447cc49583b6aaf829230c2575a15d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 podman[100591]: 2024-09-20 19:56:00.503638591 +0000 UTC m=+0.354304830 container start 20c10837ba0d06b0d675a8b6a4c40dd47447cc49583b6aaf829230c2575a15d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: pidfile_write: ignore empty --pid-file 2024-09-20T19:56:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: load: jerasure load: lrc 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Git sha 0 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: DB SUMMARY 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: DB Session ID: WDERKABWYBAWAXSM5UIL 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: MANIFEST file: MANIFEST-000049 size: 307 Bytes 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 2, files: 000044.sst 000047.sst 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000048.log size: 2453242 ; 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:56:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.env: 0x562533ee3120 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.info_log: 0x562534c8f700 2024-09-20T19:56:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:56:00.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.statistics: (nil) 2024-09-20T19:56:00.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.use_fsync: 0 2024-09-20T19:56:00.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:56:00.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.db_log_dir: 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.wal_dir: 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:56:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.write_buffer_manager: 0x562534c921e0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:56:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.unordered_write: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.row_cache: None 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.wal_filter: None 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.wal_compression: 0 2024-09-20T19:56:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:56:00.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_open_files: -1 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:56:00.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Compression algorithms supported: 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kZSTD supported: 0 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:56:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000049 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.merge_operator: 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_filter: None 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562534c8f320) 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:56:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_type: 0 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_index_type: 0 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_shortening: 1 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: checksum: 4 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: no_block_cache: 0 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache: 0x562534c83610 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_options: 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: capacity : 536870912 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_shard_bits : 4 2024-09-20T19:56:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout: strict_capacity_limit : 0 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_compressed: (nil) 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: persistent_cache: (nil) 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size: 4096 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size_deviation: 10 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_restart_interval: 16 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_block_restart_interval: 1 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: metadata_block_size: 4096 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: partition_filters: 0 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: use_delta_encoding: 1 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: filter_policy: bloomfilter 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: whole_key_filtering: 1 2024-09-20T19:56:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout: verify_compression: 0 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: format_version: 5 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_index_compression: 1 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_align: 0 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: prepopulate_block_cache: 0 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression: NoCompression 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:56:00.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.num_levels: 7 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:56:00.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:56:00.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:56:00.815 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:56:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:56:00.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.ttl: 2592000 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:56:00.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000049 succeeded,manifest_file_number is 49, next_file_number is 51, last_sequence is 21911, log_number is 43,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 43 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 43 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862160531146, "job": 1, "event": "recovery_started", "wal_files": [48]} 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #48 mode 2 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862160540250, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 52, "file_size": 2312270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 21914, "largest_seqno": 22844, "table_properties": {"data_size": 2307295, "index_size": 2651, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1285, "raw_key_size": 12143, "raw_average_key_size": 23, "raw_value_size": 2297324, "raw_average_value_size": 4531, "num_data_blocks": 118, "num_entries": 507, "num_filter_entries": 507, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862160, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dae6ac00-53f0-46ca-a668-1ae1bf7dd4c3", "db_session_id": "WDERKABWYBAWAXSM5UIL", "orig_file_number": 52, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862160540400, "job": 1, "event": "recovery_finished"} 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:5047] Creating manifest 54 2024-09-20T19:56:00.819 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000048.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562534cb2e00 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: DB pointer 0x562534db4000 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** DB Stats ** 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:00.820 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: L0 2/0 2.65 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 569.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: L6 1/0 8.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: Sum 3/0 11.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 569.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 569.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:00.821 INFO:journalctl@ceph.mon.a.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 569.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:56:00.822 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative compaction: 0.00 GB write, 161.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval compaction: 0.00 GB write, 161.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache BinnedLRUCache@0x562534c83610#2 capacity: 512.00 MB usage: 6.34 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.4e-05 secs_since: 0 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,2.31 KB,0.000441074%) IndexBlock(2,4.03 KB,0.0007689%) Misc(1,0.00 KB,0%) 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: starting mon.a rank 0 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???) e3 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).mds e1 new map 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).mds e1 print_map 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: e1 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T19:56:00.823 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout: legacy client fscid: -1 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout: No filesystems configured 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:00.824 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 ceph-mon[100622]: mon.a@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-09-20T19:56:00.825 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 bash[100591]: 20c10837ba0d06b0d675a8b6a4c40dd47447cc49583b6aaf829230c2575a15d8 2024-09-20T19:56:00.825 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:00 smithi184 systemd[1]: Started Ceph mon.a for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:02] "GET /metrics HTTP/1.1" 200 38672 "" "Prometheus/2.43.0" 2024-09-20T19:56:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:03.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:03.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: mon.a calling monitor election 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: monmap epoch 3 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: min_mon_release 18 (reef) 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: election_strategy: 1 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: fsmap 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: mgrmap e41: y(active, since 20s), standbys: x 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:03.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: mon.a calling monitor election 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:03.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: monmap epoch 3 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: min_mon_release 18 (reef) 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: election_strategy: 1 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: fsmap 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: mgrmap e41: y(active, since 20s), standbys: x 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:03.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:03 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: mon.a calling monitor election 2024-09-20T19:56:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: monmap epoch 3 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: min_mon_release 18 (reef) 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: election_strategy: 1 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: fsmap 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: mgrmap e41: y(active, since 20s), standbys: x 2024-09-20T19:56:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='' 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:03 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:04 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:04.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:05.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:05.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:05.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:56:05.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:56:05.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T19:56:05.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: Upgrade: It appears safe to stop mon.c 2024-09-20T19:56:05.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:05 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: Upgrade: It appears safe to stop mon.c 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:56:06.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T19:56:06.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T19:56:06.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: Upgrade: It appears safe to stop mon.c 2024-09-20T19:56:06.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:05 smithi184 ceph-mon[30019]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:06.887 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:06 smithi184 ceph-mon[100622]: Health check failed: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:56:06.887 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:06 smithi184 ceph-mon[30019]: Health check failed: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:56:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:06 smithi196 ceph-mon[25662]: Health check failed: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:56:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:07 smithi196 ceph-mon[25662]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:08.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:07 smithi184 ceph-mon[100622]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:08.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:07 smithi184 ceph-mon[30019]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:10.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[100622]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:10.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:10.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:56:10.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:56:10.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:10.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[30019]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:10.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:10.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:56:10.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:56:10.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:10 smithi184 ceph-mon[30019]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:10 smithi196 ceph-mon[25662]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-09-20T19:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:10 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:10 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:10 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:10 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:56:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:11 smithi196 ceph-mon[25662]: Upgrade: Updating mon.c 2024-09-20T19:56:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:11 smithi196 ceph-mon[25662]: Deploying daemon mon.c on smithi184 2024-09-20T19:56:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:11 smithi184 ceph-mon[100622]: Upgrade: Updating mon.c 2024-09-20T19:56:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:11 smithi184 ceph-mon[100622]: Deploying daemon mon.c on smithi184 2024-09-20T19:56:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:11 smithi184 ceph-mon[30019]: Upgrade: Updating mon.c 2024-09-20T19:56:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:11 smithi184 ceph-mon[30019]: Deploying daemon mon.c on smithi184 2024-09-20T19:56:12.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[100622]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:12.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.193 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:12] "GET /metrics HTTP/1.1" 200 38672 "" "Prometheus/2.43.0" 2024-09-20T19:56:12.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[30019]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:12.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:12 smithi184 ceph-mon[30019]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:12 smithi196 ceph-mon[25662]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:12.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:12 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:12 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:12.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:12 smithi196 ceph-mon[25662]: from='mgr.34107 ' entity='mgr.y' 2024-09-20T19:56:13.386 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-mon[100622]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-20T19:56:13.386 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-mon[100622]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:13.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-mon[30019]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-20T19:56:13.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-mon[30019]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:13 smithi196 ceph-mon[25662]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-20T19:56:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:13 smithi196 ceph-mon[25662]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:13.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:13.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:13.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:13.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:13.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:13 smithi184 systemd[1]: Stopping Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c[30015]: 2024-09-20T19:56:14.457+0000 7f93affa6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c[30015]: 2024-09-20T19:56:14.457+0000 7f93affa6640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE Bus STOPPING 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE Bus STOPPED 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE Bus STARTING 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE Serving on http://:::9283 2024-09-20T19:56:14.735 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: [20/Sep/2024:19:56:14] ENGINE Bus STARTED 2024-09-20T19:56:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:14 smithi184 podman[102288]: 2024-09-20 19:56:14.862418534 +0000 UTC m=+0.747897071 container died 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T19:56:15.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:15 smithi184 podman[102288]: 2024-09-20 19:56:15.235768047 +0000 UTC m=+1.121246578 container cleanup 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T19:56:15.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:15 smithi184 bash[102288]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c 2024-09-20T19:56:16.242 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:15 smithi184 podman[102361]: 2024-09-20 19:56:15.91390396 +0000 UTC m=+1.225121897 container remove 4ca652acaa9e34e080ab9521d6a4f75a3ed610ade61fb80beb3956f2875ab4a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux ) 2024-09-20T19:56:16.242 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 13m - - 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 12m - - 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7m) 11s ago 8m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (8m) 4s ago 8m 89.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (72s) 11s ago 6m 55.4M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (62s) 4s ago 12m 448M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (115s) 11s ago 13m 491M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 unknown 11s ago 13m - 2048M 2024-09-20T19:56:16.788 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (12m) 4s ago 12m 62.3M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 87753b87c34f 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (12m) 11s ago 12m 55.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 4ca652acaa9e 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7m) 11s ago 7m 16.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7m) 4s ago 7m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (11m) 11s ago 11m 59.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (11m) 11s ago 11m 63.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (11m) 11s ago 11m 57.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (10m) 11s ago 10m 51.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (10m) 4s ago 10m 64.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (9m) 4s ago 9m 53.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (9m) 4s ago 9m 50.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (9m) 4s ago 9m 53.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (84s) 4s ago 8m 44.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6m) 11s ago 6m 78.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:56:16.789 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6m) 4s ago 6m 80.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:56:16.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6m) 11s ago 6m 83.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:56:16.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6m) 4s ago 6m 82.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:56:17.126 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:16 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-20T19:56:17.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:16 smithi184 systemd[1]: Stopped Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:17.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:16 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.c.service: Consumed 9.957s CPU time. 2024-09-20T19:56:17.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 systemd[1]: Starting Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:56:17.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:56:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:56:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T19:56:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:56:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 14, 2024-09-20T19:56:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:56:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:56:17.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:56:17.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 podman[102763]: 2024-09-20 19:56:17.252345565 +0000 UTC m=+0.017936403 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:56:17.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 podman[102763]: 2024-09-20 19:56:17.369051317 +0000 UTC m=+0.134642148 container create b6535806fb3cf93e2876b460a004ae8dce23cf62ad724e7a836b7e2ba9b832b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr" 2024-09-20T19:56:17.484 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:56:17.485 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "3/23 daemons upgraded", 2024-09-20T19:56:17.485 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T19:56:17.485 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:56:17.485 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:56:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 podman[102763]: 2024-09-20 19:56:17.563483287 +0000 UTC m=+0.329074123 container init b6535806fb3cf93e2876b460a004ae8dce23cf62ad724e7a836b7e2ba9b832b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 podman[102763]: 2024-09-20 19:56:17.566842822 +0000 UTC m=+0.332433656 container start b6535806fb3cf93e2876b460a004ae8dce23cf62ad724e7a836b7e2ba9b832b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: pidfile_write: ignore empty --pid-file 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: load: jerasure load: lrc 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Git sha 0 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: DB SUMMARY 2024-09-20T19:56:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: DB Session ID: VRDGNCPPBSHE3WH3RZ7V 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: MANIFEST file: MANIFEST-000010 size: 2388 Bytes 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000039.sst 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000037.log size: 5210332 ; 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:56:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.env: 0x5586b80f8120 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.info_log: 0x5586ba207700 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.statistics: (nil) 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.use_fsync: 0 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:56:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.db_log_dir: 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.wal_dir: 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:56:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.write_buffer_manager: 0x5586ba20a1e0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:56:17.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.unordered_write: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.row_cache: None 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.wal_filter: None 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.wal_compression: 0 2024-09-20T19:56:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:56:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_open_files: -1 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:56:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Compression algorithms supported: 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kZSTD supported: 0 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:56:17.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:56:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.merge_operator: 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_filter: None 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586ba207320) 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_type: 0 2024-09-20T19:56:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_index_type: 0 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_shortening: 1 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: checksum: 4 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: no_block_cache: 0 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache: 0x5586ba1fb610 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_options: 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: capacity : 536870912 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_shard_bits : 4 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: strict_capacity_limit : 0 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_compressed: (nil) 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: persistent_cache: (nil) 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size: 4096 2024-09-20T19:56:17.811 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size_deviation: 10 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_restart_interval: 16 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_block_restart_interval: 1 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: metadata_block_size: 4096 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: partition_filters: 0 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: use_delta_encoding: 1 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: filter_policy: bloomfilter 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: whole_key_filtering: 1 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: verify_compression: 0 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: format_version: 5 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_index_compression: 1 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_align: 0 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: max_auto_readahead_size: 262144 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: prepopulate_block_cache: 0 2024-09-20T19:56:17.812 INFO:journalctl@ceph.mon.c.smithi184.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression: NoCompression 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.num_levels: 7 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:56:17.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:56:17.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:56:17.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:56:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:56:17.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.ttl: 2592000 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:56:17.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 41, last_sequence is 22679, log_number is 37,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 37 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 37 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 88c00575-24f4-43bb-9171-52330b599b14 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862177595117, "job": 1, "event": "recovery_started", "wal_files": [37]} 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #37 mode 2 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862177607101, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 42, "file_size": 3204558, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 22684, "largest_seqno": 24692, "table_properties": {"data_size": 3196305, "index_size": 4583, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2629, "raw_key_size": 25087, "raw_average_key_size": 24, "raw_value_size": 3177178, "raw_average_value_size": 3093, "num_data_blocks": 200, "num_entries": 1027, "num_filter_entries": 1027, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862177, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "88c00575-24f4-43bb-9171-52330b599b14", "db_session_id": "VRDGNCPPBSHE3WH3RZ7V", "orig_file_number": 42, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862177607219, "job": 1, "event": "recovery_finished"} 2024-09-20T19:56:17.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:5047] Creating manifest 44 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000037.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5586ba22ae00 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: DB pointer 0x5586ba32c000 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** DB Stats ** 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:17.820 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.821 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: L0 1/0 3.06 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.8 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: L6 1/0 8.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: Sum 2/0 12.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.8 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.8 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:56:17.822 INFO:journalctl@ceph.mon.c.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 635.8 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative compaction: 0.00 GB write, 205.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval compaction: 0.00 GB write, 205.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:56:17.823 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache BinnedLRUCache@0x5586ba1fb610#2 capacity: 512.00 MB usage: 7.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.78 KB,0.000530481%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: starting mon.c rank 2 at public addrs [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] at bind addrs [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???) e3 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).mds e1 new map 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).mds e1 print_map 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: e1 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: legacy client fscid: -1 2024-09-20T19:56:17.824 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout: No filesystems configured 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 ceph-mon[102844]: mon.c@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 bash[102763]: b6535806fb3cf93e2876b460a004ae8dce23cf62ad724e7a836b7e2ba9b832b7 2024-09-20T19:56:17.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:17 smithi184 systemd[1]: Started Ceph mon.c for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:17.835 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:17.835 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] 2024-09-20T19:56:17.835 INFO:teuthology.orchestra.run.smithi184.stdout: daemon mon.a on smithi184 is in unknown state 2024-09-20T19:56:17.836 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:17.836 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.44100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.44112 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2517914512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.44124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: mon.c calling monitor election 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: mon.a calling monitor election 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: monmap epoch 3 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: min_mon_release 18 (reef) 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: election_strategy: 1 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: fsmap 2024-09-20T19:56:18.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: mgrmap e41: y(active, since 34s), standbys: x 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: daemon mon.a on smithi184 is in unknown state 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.44100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.44112 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2517914512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.44124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: mon.c calling monitor election 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: mon.a calling monitor election 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: monmap epoch 3 2024-09-20T19:56:18.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: min_mon_release 18 (reef) 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: election_strategy: 1 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: fsmap 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: mgrmap e41: y(active, since 34s), standbys: x 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: daemon mon.a on smithi184 is in unknown state 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1903539026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1903539026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:18.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T19:56:19.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:19.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:56:19.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.44100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:19.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.44106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.44112 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/2517914512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.44124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: mon.c calling monitor election 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: mon.a calling monitor election 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: monmap epoch 3 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: last_changed 2024-09-20T19:43:50.434988+0000 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: min_mon_release 18 (reef) 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: election_strategy: 1 2024-09-20T19:56:19.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: fsmap 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: mgrmap e41: y(active, since 34s), standbys: x 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state'] 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: daemon mon.a on smithi184 is in unknown state 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:19.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='client.? 172.21.15.184:0/1903539026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:18 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']) 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.932 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:19.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:19.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:19.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mon.a on smithi184 is in unknown state']) 2024-09-20T19:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:19 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:20.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:20.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:21.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:20 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:21 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:22] "GET /metrics HTTP/1.1" 200 38664 "" "Prometheus/2.43.0" 2024-09-20T19:56:22.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:56:22.618+0000 7fef66d8f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-20T19:56:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:22.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:22 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:23.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:23.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:23.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:23 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:24 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:25.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:25 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:26 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:27 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:28.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:28 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:29 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:30 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:31.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:31 smithi184 ceph-mon[100622]: Upgrade: It appears safe to stop mon.b 2024-09-20T19:56:31.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:31.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:31 smithi184 ceph-mon[102844]: Upgrade: It appears safe to stop mon.b 2024-09-20T19:56:32.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:31 smithi196 ceph-mon[25662]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:32.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:31 smithi196 ceph-mon[25662]: Upgrade: It appears safe to stop mon.b 2024-09-20T19:56:32.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:32] "GET /metrics HTTP/1.1" 200 38596 "" "Prometheus/2.43.0" 2024-09-20T19:56:32.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: Upgrade: Updating mon.b 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: Deploying daemon mon.b on smithi196 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:32.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: Upgrade: Updating mon.b 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: Deploying daemon mon.b on smithi196 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: Upgrade: Updating mon.b 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: Deploying daemon mon.b on smithi196 2024-09-20T19:56:33.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:33.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:33.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:32 smithi196 ceph-mon[25662]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:33 smithi196 systemd[1]: Stopping Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:56:33.714 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:33.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:33.714 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:33.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:34 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b[25658]: 2024-09-20T19:56:34.467+0000 7f0d97ad8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:56:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:34 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b[25658]: 2024-09-20T19:56:34.467+0000 7f0d97ad8640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T19:56:34.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:34 smithi196 podman[76185]: 2024-09-20 19:56:34.600942771 +0000 UTC m=+0.262360287 container died 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d) 2024-09-20T19:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:34 smithi196 podman[76185]: 2024-09-20 19:56:34.984484264 +0000 UTC m=+0.645901771 container cleanup 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-09-20T19:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:34 smithi196 bash[76185]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b 2024-09-20T19:56:35.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 podman[76197]: 2024-09-20 19:56:35.227613665 +0000 UTC m=+0.626759362 container remove 87753b87c34f4af1600a4e451bd8f8c8b02e74d90911a6f46d8e5c5cbb912614 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T19:56:35.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-20T19:56:35.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 systemd[1]: Stopped Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:35.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@mon.b.service: Consumed 12.032s CPU time. 2024-09-20T19:56:35.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 systemd[1]: Starting Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:56:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 podman[76329]: 2024-09-20 19:56:35.686345817 +0000 UTC m=+0.016335535 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:56:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:35 smithi196 podman[76329]: 2024-09-20 19:56:35.795703314 +0000 UTC m=+0.125693069 container create f9e3f0fb6c5e15281562dd3c35155857035ad78dfc51c9e9995ef36da9298a26 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 podman[76329]: 2024-09-20 19:56:36.006135556 +0000 UTC m=+0.336125269 container init f9e3f0fb6c5e15281562dd3c35155857035ad78dfc51c9e9995ef36da9298a26 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 podman[76329]: 2024-09-20 19:56:36.009370249 +0000 UTC m=+0.339359962 container start f9e3f0fb6c5e15281562dd3c35155857035ad78dfc51c9e9995ef36da9298a26 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: pidfile_write: ignore empty --pid-file 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: load: jerasure load: lrc 2024-09-20T19:56:36.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Git sha 0 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: DB SUMMARY 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: DB Session ID: 7J9LDV80FUELD6TVC0M9 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: MANIFEST file: MANIFEST-000010 size: 2632 Bytes 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000042.sst 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000040.log size: 1445616 ; 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:56:36.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:56:36.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.env: 0x56084ee6d120 2024-09-20T19:56:36.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:56:36.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.info_log: 0x56085102d700 2024-09-20T19:56:36.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:56:36.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.statistics: (nil) 2024-09-20T19:56:36.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.use_fsync: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.db_log_dir: 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.wal_dir: 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:56:36.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.write_buffer_manager: 0x5608510301e0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:56:36.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.unordered_write: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.row_cache: None 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.wal_filter: None 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.wal_compression: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:56:36.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:56:36.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_open_files: -1 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Compression algorithms supported: 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kZSTD supported: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:56:36.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.merge_operator: 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_filter: None 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:56:36.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56085102d320) 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x560851021610 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:56:36.297 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-09-20T19:56:36.298 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression: NoCompression 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:56:36.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.num_levels: 7 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:56:36.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:56:36.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:56:36.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:56:36.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.ttl: 2592000 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:56:36.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 44, last_sequence is 26052, log_number is 40,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 40 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 40 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46b47607-e812-4bb9-9259-6aefaf96e6d2 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862196036898, "job": 1, "event": "recovery_started", "wal_files": [40]} 2024-09-20T19:56:36.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #40 mode 2 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862196042125, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 45, "file_size": 923439, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 26057, "largest_seqno": 27706, "table_properties": {"data_size": 917628, "index_size": 2145, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2629, "raw_key_size": 23339, "raw_average_key_size": 22, "raw_value_size": 901391, "raw_average_value_size": 862, "num_data_blocks": 85, "num_entries": 1045, "num_filter_entries": 1045, "num_deletions": 252, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862196, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46b47607-e812-4bb9-9259-6aefaf96e6d2", "db_session_id": "7J9LDV80FUELD6TVC0M9", "orig_file_number": 45, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862196042271, "job": 1, "event": "recovery_finished"} 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:5047] Creating manifest 47 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000040.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560851050e00 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: DB pointer 0x560851152000 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:36.306 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 901.80 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 323.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: L6 1/0 11.00 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 2/0 11.88 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 323.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 323.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-09-20T19:56:36.307 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 323.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 116.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 116.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:56:36.308 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x560851021610#2 capacity: 512.00 MB usage: 5.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.78 KB,0.000530481%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: starting mon.b rank 1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???) e3 preinit fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).mds e1 new map 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).mds e1 print_map 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-09-20T19:56:36.309 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 ceph-mon[76349]: mon.b@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 bash[76329]: f9e3f0fb6c5e15281562dd3c35155857035ad78dfc51c9e9995ef36da9298a26 2024-09-20T19:56:36.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:36 smithi196 systemd[1]: Started Ceph mon.b for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:56:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:37 smithi196 ceph-mon[76349]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:37 smithi184 ceph-mon[100622]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:37 smithi184 ceph-mon[102844]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:41.548 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:56:41.054+0000 7fef66d8f640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-20T19:56:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:42] "GET /metrics HTTP/1.1" 200 38596 "" "Prometheus/2.43.0" 2024-09-20T19:56:43.625 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:43.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:43.626 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:43.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mon.b calling monitor election 2024-09-20T19:56:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mon.c calling monitor election 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mon.b calling monitor election 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: monmap epoch 4 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: min_mon_release 19 (squid) 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: election_strategy: 1 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:43.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: fsmap 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mon.a calling monitor election 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: monmap epoch 4 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:43.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: min_mon_release 19 (squid) 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: election_strategy: 1 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: fsmap 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:43.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Reconfiguring mon.a (monmap changed)... 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:43.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:43 smithi196 ceph-mon[76349]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:56:44.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mon.b calling monitor election 2024-09-20T19:56:44.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mon.c calling monitor election 2024-09-20T19:56:44.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mon.b calling monitor election 2024-09-20T19:56:44.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: monmap epoch 4 2024-09-20T19:56:44.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: min_mon_release 19 (squid) 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: election_strategy: 1 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: fsmap 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mon.a calling monitor election 2024-09-20T19:56:44.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: monmap epoch 4 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: min_mon_release 19 (squid) 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: election_strategy: 1 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:44.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: fsmap 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:44.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:44.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mon.b calling monitor election 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mon.c calling monitor election 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mon.b calling monitor election 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: monmap epoch 4 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:44.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: min_mon_release 19 (squid) 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: election_strategy: 1 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: fsmap 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:44.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mon.a calling monitor election 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: monmap epoch 4 2024-09-20T19:56:44.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: fsid 4b598dd2-7788-11ef-baf5-efdc52797490 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: last_changed 2024-09-20T19:56:37.594077+0000 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: created 2024-09-20T19:42:15.806713+0000 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: min_mon_release 19 (squid) 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: election_strategy: 1 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2024-09-20T19:56:44.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: 2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: fsmap 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: mgrmap e41: y(active, since 59s), standbys: x 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:56:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Reconfiguring mon.a (monmap changed)... 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:44.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[100622]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Reconfiguring mon.a (monmap changed)... 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:44.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:44.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:43 smithi184 ceph-mon[102844]: Reconfiguring daemon mon.a on smithi184 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:56:44.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[100622]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:44 smithi184 ceph-mon[102844]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:44 smithi196 ceph-mon[76349]: Reconfiguring daemon mgr.y on smithi184 2024-09-20T19:56:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:47.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:56:47.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:56:47.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:47.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:56:47.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:47.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:56:47.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:56:47.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:47.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:56:47.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: Reconfiguring mon.c (monmap changed)... 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: Reconfiguring daemon mon.c on smithi184 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:56:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:48.125 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:56:48.126 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:48 smithi184 ceph-mon[100622]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T19:56:48.126 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:48 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.0 on smithi184 2024-09-20T19:56:48.126 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:48 smithi184 ceph-mon[102844]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T19:56:48.126 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:48 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.0 on smithi184 2024-09-20T19:56:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:48 smithi196 ceph-mon[76349]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T19:56:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:48 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.0 on smithi184 2024-09-20T19:56:48.696 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:56:48.696 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 13m - - 2024-09-20T19:56:48.696 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 13m - - 2024-09-20T19:56:48.696 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7m) 18s ago 8m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:56:48.696 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (8m) 16s ago 8m 89.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (104s) 18s ago 6m 56.3M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (94s) 16s ago 12m 449M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2m) 18s ago 14m 496M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (48s) 18s ago 14m 43.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 starting - - - 2048M 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (31s) 18s ago 12m 40.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (8m) 18s ago 8m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (8m) 16s ago 8m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (12m) 18s ago 12m 60.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 292435ba66e3 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (11m) 18s ago 11m 63.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (11m) 18s ago 11m 57.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (11m) 18s ago 11m 51.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (10m) 16s ago 10m 64.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (10m) 16s ago 10m 53.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:56:48.697 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (10m) 16s ago 10m 50.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (9m) 16s ago 9m 53.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (116s) 16s ago 9m 44.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7m) 18s ago 7m 79.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7m) 16s ago 7m 80.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6m) 18s ago 6m 84.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:56:48.698 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6m) 16s ago 6m 82.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:56:49.015 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:56:49.016 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:56:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 12, 2024-09-20T19:56:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 5 2024-09-20T19:56:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:56:49.017 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:56:49.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:49.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:49.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T19:56:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:56:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.1 on smithi184 2024-09-20T19:56:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/775994689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:49.292 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr" 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "4/23 daemons upgraded", 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:56:49.293 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:56:49.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:49.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:49.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.1 on smithi184 2024-09-20T19:56:49.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/775994689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T19:56:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:56:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.1 on smithi184 2024-09-20T19:56:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/775994689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:56:49.611 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:49.611 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:49.612 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:56:50.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3002908203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3002908203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:50.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:56:50.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:50.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3002908203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:56:50.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.2 on smithi184 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.2 on smithi184 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:56:51.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.2 on smithi184 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:56:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:52.154 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:56:52] "GET /metrics HTTP/1.1" 200 38664 "" "Prometheus/2.43.0" 2024-09-20T19:56:52.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:52 smithi196 ceph-mon[76349]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T19:56:52.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:52 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.3 on smithi184 2024-09-20T19:56:52.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[100622]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T19:56:52.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.3 on smithi184 2024-09-20T19:56:52.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[102844]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T19:56:52.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.3 on smithi184 2024-09-20T19:56:52.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:52.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:53.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:53 smithi196 ceph-mon[76349]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:53 smithi184 ceph-mon[100622]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:53.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:53 smithi184 ceph-mon[102844]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:53.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:56:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:56:53.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:56:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: Reconfiguring rgw.foo.smithi184.ctohee (monmap changed)... 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:54 smithi196 ceph-mon[76349]: Reconfiguring daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: Reconfiguring rgw.foo.smithi184.ctohee (monmap changed)... 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[100622]: Reconfiguring daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:56:55.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: Reconfiguring rgw.foo.smithi184.ctohee (monmap changed)... 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:55.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:54 smithi184 ceph-mon[102844]: Reconfiguring daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T19:56:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:56.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:56:57.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:56 smithi196 ceph-mon[76349]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:57.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:57.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[100622]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[102844]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:56:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:57 smithi196 ceph-mon[76349]: Reconfiguring rgw.smpl.smithi184.fllvrf (monmap changed)... 2024-09-20T19:56:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:57 smithi196 ceph-mon[76349]: Reconfiguring daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:56:57.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:57 smithi184 ceph-mon[100622]: Reconfiguring rgw.smpl.smithi184.fllvrf (monmap changed)... 2024-09-20T19:56:57.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:57 smithi184 ceph-mon[100622]: Reconfiguring daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:56:57.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:57 smithi184 ceph-mon[102844]: Reconfiguring rgw.smpl.smithi184.fllvrf (monmap changed)... 2024-09-20T19:56:57.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:57 smithi184 ceph-mon[102844]: Reconfiguring daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T19:56:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: Reconfiguring mon.b (monmap changed)... 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T19:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:56:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: Reconfiguring daemon mon.b on smithi196 2024-09-20T19:56:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:56:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:56:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[102844]: Reconfiguring daemon mgr.x on smithi196 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T19:57:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:57:01.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:57:01.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:01.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:00 smithi184 ceph-mon[100622]: Reconfiguring daemon mgr.x on smithi196 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:01.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:00 smithi196 ceph-mon[76349]: Reconfiguring daemon mgr.x on smithi196 2024-09-20T19:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:02] "GET /metrics HTTP/1.1" 200 38663 "" "Prometheus/2.43.0" 2024-09-20T19:57:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.4 on smithi196 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:57:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:02 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.4 on smithi196 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:02.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:02 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.4 on smithi196 2024-09-20T19:57:03.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:03 smithi196 ceph-mon[76349]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:03.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:57:03.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[100622]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[102844]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:03.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:03.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T19:57:03.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:03.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:03.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T19:57:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.5 on smithi196 2024-09-20T19:57:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:57:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T19:57:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.5 on smithi196 2024-09-20T19:57:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.5 on smithi196 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T19:57:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:05 smithi196 ceph-mon[76349]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:05 smithi196 ceph-mon[76349]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T19:57:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:05 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.6 on smithi196 2024-09-20T19:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[100622]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[100622]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T19:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.6 on smithi196 2024-09-20T19:57:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[102844]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[102844]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T19:57:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:05 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.6 on smithi196 2024-09-20T19:57:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:57:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T19:57:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: Reconfiguring daemon osd.7 on smithi196 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:07.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T19:57:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: Reconfiguring daemon osd.7 on smithi196 2024-09-20T19:57:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: Reconfiguring daemon osd.7 on smithi196 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:08 smithi196 ceph-mon[76349]: Reconfiguring rgw.foo.smithi196.zyivrs (monmap changed)... 2024-09-20T19:57:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:08 smithi196 ceph-mon[76349]: Reconfiguring daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:57:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:08 smithi184 ceph-mon[100622]: Reconfiguring rgw.foo.smithi196.zyivrs (monmap changed)... 2024-09-20T19:57:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:08 smithi184 ceph-mon[100622]: Reconfiguring daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:57:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:08 smithi184 ceph-mon[102844]: Reconfiguring rgw.foo.smithi196.zyivrs (monmap changed)... 2024-09-20T19:57:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:08 smithi184 ceph-mon[102844]: Reconfiguring daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T19:57:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: Reconfiguring rgw.smpl.smithi196.vsmvti (monmap changed)... 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[100622]: Reconfiguring daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: Reconfiguring rgw.smpl.smithi196.vsmvti (monmap changed)... 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:09 smithi184 ceph-mon[102844]: Reconfiguring daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: Reconfiguring rgw.smpl.smithi196.vsmvti (monmap changed)... 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:09 smithi196 ceph-mon[76349]: Reconfiguring daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: Upgrade: Setting container_image for all mon 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T19:57:10.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: Upgrade: Setting container_image for all crash 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[100622]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: Upgrade: Setting container_image for all mon 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T19:57:10.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: Upgrade: Setting container_image for all crash 2024-09-20T19:57:10.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:10.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:10.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:10 smithi184 ceph-mon[102844]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: Upgrade: Setting container_image for all mon 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T19:57:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: Upgrade: Setting container_image for all crash 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:10 smithi196 ceph-mon[76349]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: Upgrade: Updating osd.0 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:11 smithi196 ceph-mon[76349]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:12] "GET /metrics HTTP/1.1" 200 38663 "" "Prometheus/2.43.0" 2024-09-20T19:57:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: Upgrade: Updating osd.0 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[100622]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: Upgrade: Updating osd.0 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:12.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:11 smithi184 ceph-mon[102844]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:13.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:13 smithi184 ceph-mon[100622]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:13.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:13 smithi184 ceph-mon[102844]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:13 smithi196 ceph-mon[76349]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:13.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:13.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:14.069 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:14 smithi184 systemd[1]: Stopping Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:57:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:15.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-mon[100622]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:15.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-mon[100622]: osd.0 marked itself down and dead 2024-09-20T19:57:15.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-mon[102844]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:15.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-mon[102844]: osd.0 marked itself down and dead 2024-09-20T19:57:15.274 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[34428]: 2024-09-20T19:57:15.039+0000 7f55088fb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:57:15.274 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[34428]: 2024-09-20T19:57:15.039+0000 7f55088fb640 -1 osd.0 106 *** Got signal Terminated *** 2024-09-20T19:57:15.274 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[34428]: 2024-09-20T19:57:15.039+0000 7f55088fb640 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:57:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:15 smithi196 ceph-mon[76349]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:15 smithi196 ceph-mon[76349]: osd.0 marked itself down and dead 2024-09-20T19:57:15.548 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 podman[108102]: 2024-09-20 19:57:15.406558821 +0000 UTC m=+0.488456445 container died 292435ba66e32c73017f3d15314f4d81d63962d312375c64473beba84cea39d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-20T19:57:16.041 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 podman[108102]: 2024-09-20 19:57:15.797818214 +0000 UTC m=+0.879715833 container cleanup 292435ba66e32c73017f3d15314f4d81d63962d312375c64473beba84cea39d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T19:57:16.041 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:15 smithi184 bash[108102]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:57:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[102844]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T19:57:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:16.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:16.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[100622]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T19:57:16.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:16.300 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108117]: 2024-09-20 19:57:16.040662799 +0000 UTC m=+0.631031733 container remove 292435ba66e32c73017f3d15314f4d81d63962d312375c64473beba84cea39d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:57:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:16 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:16 smithi196 ceph-mon[76349]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T19:57:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:16.654 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.281160209 +0000 UTC m=+0.015845646 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:16.654 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.408675824 +0000 UTC m=+0.143361259 container create 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T19:57:16.654 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.652792071 +0000 UTC m=+0.387477505 container init 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T19:57:17.048 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.656264564 +0000 UTC m=+0.390949992 container start 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:57:17.048 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.761497941 +0000 UTC m=+0.496183378 container attach 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T19:57:17.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:16 smithi184 podman[108208]: 2024-09-20 19:57:16.840786448 +0000 UTC m=+0.575471895 container died 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T19:57:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:17 smithi196 ceph-mon[76349]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:17 smithi196 ceph-mon[76349]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T19:57:17.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:17 smithi184 ceph-mon[100622]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:17.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:17 smithi184 ceph-mon[100622]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T19:57:17.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:17 smithi184 ceph-mon[102844]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:17.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:17 smithi184 ceph-mon[102844]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T19:57:17.798 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 podman[108230]: 2024-09-20 19:57:17.473670059 +0000 UTC m=+0.623063567 container remove 135159f95b51beb4d29ede4eed66b6f1b8ba7d3f8024dc04ebfe27ecf3b4de89 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T19:57:17.798 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T19:57:17.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 systemd[1]: Stopped Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:57:17.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.0.service: Consumed 6.932s CPU time. 2024-09-20T19:57:17.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 systemd[1]: Starting Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:57:18.298 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 podman[108327]: 2024-09-20 19:57:17.817431677 +0000 UTC m=+0.015979869 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:18.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:17 smithi184 podman[108327]: 2024-09-20 19:57:17.924916107 +0000 UTC m=+0.123464285 container create 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T19:57:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[102844]: pgmap v54: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-mon[100622]: pgmap v54: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:18.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 podman[108327]: 2024-09-20 19:57:18.302590379 +0000 UTC m=+0.501138564 container init 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T19:57:18.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 podman[108327]: 2024-09-20 19:57:18.306086019 +0000 UTC m=+0.504634198 container start 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T19:57:18.800 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 podman[108327]: 2024-09-20 19:57:18.411091125 +0000 UTC m=+0.609639306 container attach 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T19:57:18.800 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:18.800 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 bash[108327]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:18.800 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:18.800 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 bash[108327]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:18 smithi196 ceph-mon[76349]: pgmap v54: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 bash[108327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:18 smithi184 bash[108327]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:19.196 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 bash[108327]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 bash[108327]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 bash[108327]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 bash[108327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:19.533 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[108342]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T19:57:19.534 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 bash[108327]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T19:57:19.534 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 podman[108327]: 2024-09-20 19:57:19.224298046 +0000 UTC m=+1.422846225 container died 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-20T19:57:19.920 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:57:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:19 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:20.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:19 smithi184 podman[108481]: 2024-09-20 19:57:19.815040019 +0000 UTC m=+0.584470983 container remove 3009b2e8463b3c29da916ac7b9a61a61611489b8a578cce096fdffb630e562e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:57:20.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 podman[108563]: 2024-09-20 19:57:19.931466179 +0000 UTC m=+0.017101816 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:19 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:20.311 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 podman[108563]: 2024-09-20 19:57:20.049301795 +0000 UTC m=+0.134937420 container create 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T19:57:20.311 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 podman[108563]: 2024-09-20 19:57:20.285308342 +0000 UTC m=+0.370943972 container init 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T19:57:20.311 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 podman[108563]: 2024-09-20 19:57:20.288679219 +0000 UTC m=+0.374314845 container start 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-20T19:57:20.311 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-osd[108626]: -- 172.21.15.184:0/3962458280 <== mon.2 v2:172.21.15.184:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5568d4f79a40 con 0x5568d5d52400 2024-09-20T19:57:20.484 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:57:20.484 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 14m - - 2024-09-20T19:57:20.484 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 13m - - 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (8m) 9s ago 9m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (9m) 3s ago 9m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2m) 9s ago 7m 58.1M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2m) 3s ago 13m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2m) 9s ago 14m 501M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (80s) 9s ago 15m 46.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (44s) 3s ago 13m 35.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (63s) 9s ago 13m 46.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (8m) 9s ago 8m 16.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (8m) 3s ago 8m 16.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 starting - - - 4096M 2024-09-20T19:57:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (12m) 9s ago 12m 63.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (12m) 9s ago 12m 57.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (11m) 9s ago 11m 51.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (11m) 3s ago 11m 64.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (10m) 3s ago 10m 53.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (10m) 3s ago 10m 51.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (10m) 3s ago 10m 54.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2m) 3s ago 9m 49.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7m) 9s ago 7m 79.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7m) 3s ago 7m 81.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:57:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7m) 9s ago 7m 84.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:57:20.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7m) 3s ago 7m 83.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:57:20.632 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 bash[108563]: 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 2024-09-20T19:57:20.632 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:20 smithi184 systemd[1]: Started Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:20.855 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 11, 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 5 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:57:20.856 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: pgmap v55: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: pgmap v55: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:20.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:20.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: pgmap v55: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:21.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:57:21.161 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:57:21.531 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded 2024-09-20T19:57:21.531 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:57:21.531 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: osd.0 (root=default,host=smithi184) is down 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-20T19:57:21.532 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-09-20T19:57:21.533 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1d is active+undersized+degraded, acting [1,4] 2024-09-20T19:57:22.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:21.950+0000 7fd0cf98e640 -1 Falling back to public interface 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='client.54152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/293833973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='client.54152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/293833973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2428532427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2428532427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='client.54152 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/293833973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2428532427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:22] "GET /metrics HTTP/1.1" 200 38659 "" "Prometheus/2.43.0" 2024-09-20T19:57:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: pgmap v56: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: pgmap v56: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: pgmap v56: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.696 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:23.523Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:23.697 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:23.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:23.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:23.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:24.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:24.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:25.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:25.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:25.548 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:25.522+0000 7fd0cf98e640 -1 osd.0 106 log_to_monitors true 2024-09-20T19:57:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:26.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:26.779 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:26.521+0000 7fd0c7738640 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:27.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/636 objects degraded (10.535%) 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:27.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639] boot 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639] boot 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: osd.0 [v2:172.21.15.184:6802/3790958639,v1:172.21.15.184:6803/3790958639] boot 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: pgmap v61: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: pgmap v61: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: pgmap v61: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.199 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.199 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:30.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: pgmap v63: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/636 objects degraded (4.560%) 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: pgmap v63: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/636 objects degraded (4.560%) 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: pgmap v63: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 29/636 objects degraded (4.560%) 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:31.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:31.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:31.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:31.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:32] "GET /metrics HTTP/1.1" 200 38586 "" "Prometheus/2.43.0" 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[100622]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:32.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:31 smithi184 ceph-mon[102844]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T19:57:32.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:31 smithi196 ceph-mon[76349]: Upgrade: osd.0 is safe to restart 2024-09-20T19:57:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:32 smithi184 ceph-mon[100622]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:32 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded) 2024-09-20T19:57:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:32 smithi184 ceph-mon[102844]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:32 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded) 2024-09-20T19:57:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:32 smithi196 ceph-mon[76349]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:32 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/636 objects degraded (4.560%), 8 pgs degraded) 2024-09-20T19:57:33.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:33.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:33.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:33.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[100622]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[100622]: Upgrade: Updating osd.0 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[100622]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:34.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[102844]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T19:57:34.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[102844]: Upgrade: Updating osd.0 2024-09-20T19:57:34.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:34.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:34.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:34 smithi184 ceph-mon[102844]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:34 smithi196 ceph-mon[76349]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T19:57:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:34 smithi196 ceph-mon[76349]: Upgrade: Updating osd.0 2024-09-20T19:57:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T19:57:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:34 smithi196 ceph-mon[76349]: Deploying daemon osd.0 on smithi184 2024-09-20T19:57:35.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:35.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:36.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:37.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[100622]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:37.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[102844]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:37.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:36 smithi196 ceph-mon[76349]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.433 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:38 smithi184 systemd[1]: Stopping Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:57:38.433 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:38.370+0000 7fd0cc926640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:57:38.433 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:38.370+0000 7fd0cc926640 -1 osd.0 111 *** Got signal Terminated *** 2024-09-20T19:57:38.433 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[108600]: 2024-09-20T19:57:38.370+0000 7fd0cc926640 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:57:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[100622]: osd.0 marked itself down and dead 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:38 smithi184 ceph-mon[102844]: osd.0 marked itself down and dead 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:57:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:38 smithi196 ceph-mon[76349]: osd.0 marked itself down and dead 2024-09-20T19:57:39.298 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:39 smithi184 podman[112473]: 2024-09-20 19:57:39.034825023 +0000 UTC m=+0.755438355 container died 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T19:57:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[100622]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T19:57:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[102844]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T19:57:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:39.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:39 smithi184 podman[112473]: 2024-09-20 19:57:39.361319689 +0000 UTC m=+1.081933019 container cleanup 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T19:57:39.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:39 smithi184 bash[112473]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0 2024-09-20T19:57:39.799 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:39 smithi184 podman[112487]: 2024-09-20 19:57:39.546005639 +0000 UTC m=+0.507229463 container remove 38492abfdd0f6325588a8d87a812dd6361bedbfee319ff32bfa5ccda5f04e458 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:57:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:39 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:57:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:39 smithi196 ceph-mon[76349]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T19:57:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:40.229 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:39 smithi184 podman[112581]: 2024-09-20 19:57:39.877495175 +0000 UTC m=+0.016119726 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:40.317 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:40 smithi184 podman[112581]: 2024-09-20 19:57:40.00524448 +0000 UTC m=+0.143869017 container create 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T19:57:40.318 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:40 smithi184 podman[112581]: 2024-09-20 19:57:40.224765637 +0000 UTC m=+0.363390181 container init 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:57:40.318 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:40 smithi184 podman[112581]: 2024-09-20 19:57:40.22808018 +0000 UTC m=+0.366704724 container start 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:57:40.548 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:40 smithi184 podman[112581]: 2024-09-20 19:57:40.324849979 +0000 UTC m=+0.463474514 container attach 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T19:57:40.548 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:40 smithi184 podman[112581]: 2024-09-20 19:57:40.416225374 +0000 UTC m=+0.554849925 container died 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:40 smithi196 ceph-mon[76349]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T19:57:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:40 smithi196 ceph-mon[76349]: pgmap v70: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:41.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:40 smithi184 ceph-mon[100622]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T19:57:41.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:40 smithi184 ceph-mon[100622]: pgmap v70: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:41.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:40 smithi184 ceph-mon[102844]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T19:57:41.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:40 smithi184 ceph-mon[102844]: pgmap v70: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:41.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112604]: 2024-09-20 19:57:41.003689908 +0000 UTC m=+0.581617308 container remove 07578f0af65e46abf905fc99cfcbefa6ed86d00009bd098be51c3d0accf29891 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-deactivate, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:57:41.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T19:57:41.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 systemd[1]: Stopped Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:57:41.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.0.service: Consumed 1.778s CPU time. 2024-09-20T19:57:41.299 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 systemd[1]: Starting Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:57:41.709 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112703]: 2024-09-20 19:57:41.346166383 +0000 UTC m=+0.015704042 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:41.709 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112703]: 2024-09-20 19:57:41.46355986 +0000 UTC m=+0.133097507 container create db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T19:57:41.709 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112703]: 2024-09-20 19:57:41.708036411 +0000 UTC m=+0.377574066 container init db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T19:57:42.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112703]: 2024-09-20 19:57:41.711824261 +0000 UTC m=+0.381361910 container start db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:57:42.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 podman[112703]: 2024-09-20 19:57:41.791402372 +0000 UTC m=+0.460940035 container attach db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T19:57:42.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:42.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 bash[112703]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:42.003 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:42.004 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:41 smithi184 bash[112703]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:57:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:42] "GET /metrics HTTP/1.1" 200 38586 "" "Prometheus/2.43.0" 2024-09-20T19:57:42.593 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:42.593 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:42.593 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:42.593 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:42.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-mon[100622]: pgmap v71: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:42.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-mon[102844]: pgmap v71: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:42.929 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate[112718]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 bash[112703]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T19:57:42.930 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:42 smithi184 podman[112703]: 2024-09-20 19:57:42.630754614 +0000 UTC m=+1.300292277 container died db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T19:57:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:42 smithi196 ceph-mon[76349]: pgmap v71: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:57:43.497 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 podman[112848]: 2024-09-20 19:57:43.245281028 +0000 UTC m=+0.603633664 container remove db9123211102fc39c5b9351a119fdfa6dfcb34d2a188071023606bec5f880633 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T19:57:43.497 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 podman[112919]: 2024-09-20 19:57:43.360336289 +0000 UTC m=+0.015841796 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:57:43.497 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 podman[112919]: 2024-09-20 19:57:43.496288384 +0000 UTC m=+0.151793891 container create 8a6cac5c11f0d74ce7af6d14058fb5a35fe166bce0037a150b4f00952b2bd945 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD) 2024-09-20T19:57:43.767 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:43.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:43.767 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:43.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:43.767 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 podman[112919]: 2024-09-20 19:57:43.739527082 +0000 UTC m=+0.395032599 container init 8a6cac5c11f0d74ce7af6d14058fb5a35fe166bce0037a150b4f00952b2bd945 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-20T19:57:43.767 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 podman[112919]: 2024-09-20 19:57:43.742997561 +0000 UTC m=+0.398503066 container start 8a6cac5c11f0d74ce7af6d14058fb5a35fe166bce0037a150b4f00952b2bd945 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T19:57:44.018 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 ceph-osd[112937]: -- 172.21.15.184:0/3459215061 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a80ec69a40 con 0x55a80fa38000 2024-09-20T19:57:44.019 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 bash[112919]: 8a6cac5c11f0d74ce7af6d14058fb5a35fe166bce0037a150b4f00952b2bd945 2024-09-20T19:57:44.019 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:43 smithi184 systemd[1]: Started Ceph osd.0 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:57:45.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:45.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: pgmap v72: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: pgmap v72: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: pgmap v72: 161 pgs: 37 peering, 21 stale+active+clean, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.940 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[112933]: 2024-09-20T19:57:45.624+0000 7f0e243ac640 -1 Falling back to public interface 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:45.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: pgmap v73: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: pgmap v73: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:46.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:46.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:46.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: pgmap v73: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:47.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: pgmap v74: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: pgmap v74: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:48.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: pgmap v74: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.298 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[112933]: 2024-09-20T19:57:48.934+0000 7f0e243ac640 -1 osd.0 111 log_to_monitors true 2024-09-20T19:57:49.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.925 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.928 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-0[112933]: 2024-09-20T19:57:49.920+0000 7f0e1c156640 -1 osd.0 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.935 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.935 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:49.936 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:50.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: pgmap v75: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: pgmap v75: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:51.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T19:57:51.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: pgmap v75: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:57:51.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:51.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:51.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:51.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:51.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:51.848 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608] boot 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.006 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:57:52] "GET /metrics HTTP/1.1" 200 38786 "" "Prometheus/2.43.0" 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608] boot 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: osd.0 [v2:172.21.15.184:6802/3096672608,v1:172.21.15.184:6803/3096672608] boot 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T19:57:52.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:52.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 14m - - 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 14m - - 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (8m) 15s ago 9m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (9m) 14s ago 9m 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2m) 15s ago 7m 58.1M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2m) 14s ago 13m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3m) 15s ago 15m 504M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (112s) 15s ago 15m 48.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:57:52.480 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (76s) 14s ago 14m 38.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (95s) 15s ago 14m 46.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (9m) 15s ago 9m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (9m) 14s ago 9m 16.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 starting - - - 4096M 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (13m) 15s ago 13m 63.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (12m) 15s ago 12m 58.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (12m) 15s ago 12m 53.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (11m) 14s ago 11m 65.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (11m) 14s ago 11m 55.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (11m) 14s ago 11m 51.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (10m) 14s ago 10m 54.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3m) 14s ago 10m 49.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:57:52.481 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (8m) 15s ago 8m 79.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:57:52.482 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (8m) 14s ago 8m 81.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:57:52.482 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7m) 15s ago 7m 84.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:57:52.482 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7m) 14s ago 7m 83.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:57:52.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 11, 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:57:52.875 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:57:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='client.64147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:57:53.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='client.64147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='client.64147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T19:57:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T19:57:53.215 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T19:57:53.216 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:57:53.216 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T19:57:53.216 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T19:57:53.216 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:57:53.216 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-09-20T19:57:53.568 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-09-20T19:57:53.569 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1d is active+undersized+degraded, acting [1,4] 2024-09-20T19:57:53.778 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:53.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:53.778 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:57:53.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='client.64159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/150145647' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='client.64159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/150145647' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='client.54212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/534351839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:53.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='client.54212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:53.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/534351839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:53.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='client.64159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:54.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/150145647' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='client.54212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/534351839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: pgmap v80: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:54.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: pgmap v80: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: pgmap v80: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/636 objects degraded (10.535%) 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:55.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:56.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:56.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:56.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:56 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded) 2024-09-20T19:57:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:57:57.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.054 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.056 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.057 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.058 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.059 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.062 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded) 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:56 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/636 objects degraded (10.535%), 20 pgs degraded) 2024-09-20T19:57:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.814 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.815 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:57:57.819 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.820 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:57.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: Upgrade: osd.1 is safe to restart 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: Upgrade: osd.1 is safe to restart 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: Upgrade: osd.1 is safe to restart 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 0 op/s 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[102844]: Upgrade: Updating osd.1 2024-09-20T19:58:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[100622]: Upgrade: Updating osd.1 2024-09-20T19:58:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:00 smithi196 ceph-mon[76349]: Upgrade: Updating osd.1 2024-09-20T19:58:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[100622]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[100622]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[102844]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:01.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[102844]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:01.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:01 smithi196 ceph-mon[76349]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:01 smithi196 ceph-mon[76349]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:02.278 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:02] "GET /metrics HTTP/1.1" 200 38803 "" "Prometheus/2.43.0" 2024-09-20T19:58:02.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:02 smithi184 ceph-mon[100622]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:02.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:02 smithi184 ceph-mon[102844]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:02 smithi196 ceph-mon[76349]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:03.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:03.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:03.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:03.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:04.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:05.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:04 smithi184 systemd[1]: Stopping Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:58:05.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[37787]: 2024-09-20T19:58:04.945+0000 7f5898069640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:58:05.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[37787]: 2024-09-20T19:58:04.945+0000 7f5898069640 -1 osd.1 116 *** Got signal Terminated *** 2024-09-20T19:58:05.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[37787]: 2024-09-20T19:58:04.945+0000 7f5898069640 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:58:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:05 smithi196 ceph-mon[76349]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-09-20T19:58:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:05 smithi196 ceph-mon[76349]: osd.1 marked itself down and dead 2024-09-20T19:58:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:05 smithi184 ceph-mon[100622]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-09-20T19:58:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:05 smithi184 ceph-mon[100622]: osd.1 marked itself down and dead 2024-09-20T19:58:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:05 smithi184 ceph-mon[102844]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-09-20T19:58:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:05 smithi184 ceph-mon[102844]: osd.1 marked itself down and dead 2024-09-20T19:58:05.814 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:05 smithi184 podman[117534]: 2024-09-20 19:58:05.547313469 +0000 UTC m=+0.703199412 container died eb61dcd842343f92f8cbcc26e005428b0c67afc9b3cb67fb3fefcde7ede103a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T19:58:06.106 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:05 smithi184 podman[117534]: 2024-09-20 19:58:05.995715428 +0000 UTC m=+1.151601368 container cleanup eb61dcd842343f92f8cbcc26e005428b0c67afc9b3cb67fb3fefcde7ede103a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:58:06.106 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:05 smithi184 bash[117534]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1 2024-09-20T19:58:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:06 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:06 smithi196 ceph-mon[76349]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T19:58:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[100622]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T19:58:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:06.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117549]: 2024-09-20 19:58:06.314442947 +0000 UTC m=+0.761009241 container remove eb61dcd842343f92f8cbcc26e005428b0c67afc9b3cb67fb3fefcde7ede103a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T19:58:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[102844]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T19:58:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:06.910 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117636]: 2024-09-20 19:58:06.528841329 +0000 UTC m=+0.016190882 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:06.911 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117636]: 2024-09-20 19:58:06.665439833 +0000 UTC m=+0.152789373 container create 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:58:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:07 smithi184 ceph-mon[100622]: pgmap v87: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:07 smithi184 ceph-mon[100622]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T19:58:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:07 smithi184 ceph-mon[102844]: pgmap v87: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:07 smithi184 ceph-mon[102844]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T19:58:07.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117636]: 2024-09-20 19:58:06.909959943 +0000 UTC m=+0.397309486 container init 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T19:58:07.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117636]: 2024-09-20 19:58:06.913321098 +0000 UTC m=+0.400670642 container start 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T19:58:07.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:06 smithi184 podman[117636]: 2024-09-20 19:58:06.993159427 +0000 UTC m=+0.480508994 container attach 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T19:58:07.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 podman[117636]: 2024-09-20 19:58:07.107823249 +0000 UTC m=+0.595172800 container died 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T19:58:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:07 smithi196 ceph-mon[76349]: pgmap v87: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T19:58:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:07 smithi196 ceph-mon[76349]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T19:58:08.048 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 podman[117658]: 2024-09-20 19:58:07.680067985 +0000 UTC m=+0.566293322 container remove 3c5aabfa04bf94f05b17b8f6d48760cf4c45575b371ba25cd5eca04405d7d8e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T19:58:08.048 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T19:58:08.048 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 systemd[1]: Stopped Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:58:08.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.1.service: Consumed 7.783s CPU time. 2024-09-20T19:58:08.049 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:07 smithi184 systemd[1]: Starting Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:58:08.410 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 podman[117759]: 2024-09-20 19:58:08.018211798 +0000 UTC m=+0.015588196 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:08.410 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 podman[117759]: 2024-09-20 19:58:08.131127366 +0000 UTC m=+0.128503757 container create 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 podman[117759]: 2024-09-20 19:58:08.40891453 +0000 UTC m=+0.406290929 container init 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True) 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 podman[117759]: 2024-09-20 19:58:08.412554365 +0000 UTC m=+0.409930757 container start 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 podman[117759]: 2024-09-20 19:58:08.517323475 +0000 UTC m=+0.514699866 container attach 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2) 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 bash[117759]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:08.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:08 smithi184 bash[117759]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-mon[100622]: pgmap v89: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-mon[102844]: pgmap v89: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:09.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:09.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:09.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:09.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:09 smithi196 ceph-mon[76349]: pgmap v89: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:09.613 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[117774]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 bash[117759]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T19:58:09.614 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:09 smithi184 podman[117759]: 2024-09-20 19:58:09.330718801 +0000 UTC m=+1.328095196 container died 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T19:58:10.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:10 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 84/636 objects degraded (13.208%), 24 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:10.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:10 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 84/636 objects degraded (13.208%), 24 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:10.401 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 podman[117904]: 2024-09-20 19:58:10.054826024 +0000 UTC m=+0.718048992 container remove 24391424cfda343aed7902257a701fc9e8270a62ab1f7fdf97b7703a637fd18b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:58:10.401 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 podman[117961]: 2024-09-20 19:58:10.16496374 +0000 UTC m=+0.016189533 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:10.401 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 podman[117961]: 2024-09-20 19:58:10.289056193 +0000 UTC m=+0.140281966 container create 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T19:58:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:10 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 84/636 objects degraded (13.208%), 24 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:10.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 podman[117961]: 2024-09-20 19:58:10.525452543 +0000 UTC m=+0.376678322 container init 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-20T19:58:10.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 podman[117961]: 2024-09-20 19:58:10.528577747 +0000 UTC m=+0.379803519 container start 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:58:10.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 ceph-osd[117980]: -- 172.21.15.184:0/3806354097 <== mon.2 v2:172.21.15.184:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558eb292ba40 con 0x558eb2909c00 2024-09-20T19:58:10.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 bash[117961]: 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e 2024-09-20T19:58:10.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:10 smithi184 systemd[1]: Started Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: pgmap v90: 161 pgs: 32 active+undersized, 1 stale+active+clean, 24 active+undersized+degraded, 104 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84/636 objects degraded (13.208%) 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: pgmap v90: 161 pgs: 32 active+undersized, 1 stale+active+clean, 24 active+undersized+degraded, 104 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84/636 objects degraded (13.208%) 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: pgmap v90: 161 pgs: 32 active+undersized, 1 stale+active+clean, 24 active+undersized+degraded, 104 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 84/636 objects degraded (13.208%) 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:11.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:12] "GET /metrics HTTP/1.1" 200 38803 "" "Prometheus/2.43.0" 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.632 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:12.440+0000 7f9d0f4be640 -1 Falling back to public interface 2024-09-20T19:58:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:12.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: pgmap v91: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: pgmap v91: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: pgmap v91: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:13.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:13.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:13.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:13.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:13.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:13.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:13.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 19:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T19:58:13.811922319Z level=info msg="Completed cleanup jobs" duration=12.435035ms 2024-09-20T19:58:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:14.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: pgmap v92: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: pgmap v92: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: pgmap v92: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:15.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.298 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:16.004+0000 7f9d0f4be640 -1 osd.1 116 log_to_monitors true 2024-09-20T19:58:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:16.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.048 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:16 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:16.965+0000 7f9d03260640 -1 osd.1 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:58:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: pgmap v93: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: pgmap v93: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: pgmap v93: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:17.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:17.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:18.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950] boot 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950] boot 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: osd.1 [v2:172.21.15.184:6810/1256822950,v1:172.21.15.184:6811/1256822950] boot 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:18.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: pgmap v95: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: pgmap v95: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: pgmap v95: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 90/636 objects degraded (14.151%) 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:20.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: pgmap v98: 161 pgs: 17 active+undersized, 8 active+undersized+degraded, 136 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: pgmap v98: 161 pgs: 17 active+undersized, 8 active+undersized+degraded, 136 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 37/636 objects degraded (5.818%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 37/636 objects degraded (5.818%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: pgmap v98: 161 pgs: 17 active+undersized, 8 active+undersized+degraded, 136 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:58:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 37/636 objects degraded (5.818%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:21.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.145 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:22] "GET /metrics HTTP/1.1" 200 38803 "" "Prometheus/2.43.0" 2024-09-20T19:58:22.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:22.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: pgmap v99: 161 pgs: 12 active+undersized, 7 active+undersized+degraded, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: pgmap v99: 161 pgs: 12 active+undersized, 7 active+undersized+degraded, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: pgmap v99: 161 pgs: 12 active+undersized, 7 active+undersized+degraded, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:23.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:23.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:23.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:23.865 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:58:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:58:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 15m - - 2024-09-20T19:58:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 14m - - 2024-09-20T19:58:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (9m) 1s ago 10m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (10m) 6s ago 10m 89.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3m) 1s ago 8m 59.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3m) 6s ago 14m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4m) 1s ago 15m 507M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2m) 1s ago 16m 52.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:58:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (108s) 6s ago 14m 43.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2m) 1s ago 14m 50.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (9m) 1s ago 9m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (9m) 6s ago 9m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (41s) 1s ago 13m 41.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (13m) 1s ago 13m 65.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 eb61dcd84234 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (13m) 1s ago 13m 58.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:58:24.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (12m) 1s ago 12m 53.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (12m) 6s ago 12m 66.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (12m) 6s ago 12m 56.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (11m) 6s ago 11m 52.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (11m) 6s ago 11m 56.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3m) 6s ago 10m 53.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:58:24.443 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (8m) 1s ago 8m 80.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:58:24.444 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (8m) 6s ago 8m 81.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:58:24.444 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (8m) 1s ago 8m 85.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:58:24.444 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (8m) 6s ago 8m 83.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[100622]: Upgrade: osd.1 is safe to restart 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:58:24.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:24 smithi184 ceph-mon[102844]: Upgrade: osd.1 is safe to restart 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T19:58:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:24 smithi196 ceph-mon[76349]: Upgrade: osd.1 is safe to restart 2024-09-20T19:58:24.789 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:58:24.789 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:58:24.789 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:58:24.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T19:58:24.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 10, 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T19:58:24.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:58:24.792 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "6/23 daemons upgraded", 2024-09-20T19:58:25.067 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T19:58:25.068 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:58:25.068 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:58:25.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[100622]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-20T19:58:25.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[100622]: from='client.54230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[100622]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:25.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3967750597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:25.347 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[102844]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-20T19:58:25.347 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[102844]: from='client.54230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.347 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[102844]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.347 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:25.348 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3967750597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:25.393 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:25.393 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:25.393 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:25 smithi196 ceph-mon[76349]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-20T19:58:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:25 smithi196 ceph-mon[76349]: from='client.54230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:25 smithi196 ceph-mon[76349]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:25 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3967750597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:26.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[100622]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[100622]: from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4202406782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:26.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[102844]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[102844]: from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4202406782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:26 smithi196 ceph-mon[76349]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:26 smithi196 ceph-mon[76349]: from='client.44286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4202406782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[102844]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:27.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[100622]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:27.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:27 smithi196 ceph-mon[76349]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T19:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: Upgrade: Updating osd.1 2024-09-20T19:58:28.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:28.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:28.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:28.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: Upgrade: Updating osd.1 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: Upgrade: Updating osd.1 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: Deploying daemon osd.1 on smithi184 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[100622]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[102844]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:29 smithi196 ceph-mon[76349]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T19:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:31.038 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:30 smithi184 systemd[1]: Stopping Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:58:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-mon[102844]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:58:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-mon[102844]: osd.1 marked itself down and dead 2024-09-20T19:58:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-mon[100622]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:58:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-mon[100622]: osd.1 marked itself down and dead 2024-09-20T19:58:31.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:31.036+0000 7f9d0c456640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:58:31.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:31.036+0000 7f9d0c456640 -1 osd.1 121 *** Got signal Terminated *** 2024-09-20T19:58:31.299 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[117976]: 2024-09-20T19:58:31.036+0000 7f9d0c456640 -1 osd.1 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:58:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:31 smithi196 ceph-mon[76349]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T19:58:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:31 smithi196 ceph-mon[76349]: osd.1 marked itself down and dead 2024-09-20T19:58:32.003 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:31 smithi184 podman[122418]: 2024-09-20 19:58:31.751256933 +0000 UTC m=+0.824392801 container died 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-20T19:58:32.261 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:32] "GET /metrics HTTP/1.1" 200 38794 "" "Prometheus/2.43.0" 2024-09-20T19:58:32.262 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122418]: 2024-09-20 19:58:32.094429049 +0000 UTC m=+1.167564908 container cleanup 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T19:58:32.262 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 bash[122418]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1 2024-09-20T19:58:32.548 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122434]: 2024-09-20 19:58:32.387785025 +0000 UTC m=+0.636247213 container remove 8be9abb0ce829197eb19bee7ae28407da5faa8a4318c746fa7b4e5efdc2cf64e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T19:58:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[100622]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T19:58:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[100622]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T19:58:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[102844]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T19:58:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:32 smithi184 ceph-mon[102844]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T19:58:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:32 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:58:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:32 smithi196 ceph-mon[76349]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T19:58:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:32 smithi196 ceph-mon[76349]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T19:58:32.967 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122522]: 2024-09-20 19:58:32.610468397 +0000 UTC m=+0.016153561 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:32.967 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122522]: 2024-09-20 19:58:32.72200117 +0000 UTC m=+0.127686334 container create 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T19:58:33.263 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122522]: 2024-09-20 19:58:32.966353326 +0000 UTC m=+0.372038492 container init 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:58:33.263 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:32 smithi184 podman[122522]: 2024-09-20 19:58:32.970111033 +0000 UTC m=+0.375796198 container start 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T19:58:33.263 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 podman[122522]: 2024-09-20 19:58:33.07441249 +0000 UTC m=+0.480097655 container attach 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T19:58:33.263 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 podman[122522]: 2024-09-20 19:58:33.165532717 +0000 UTC m=+0.571217884 container died 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T19:58:33.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:33 smithi184 ceph-mon[100622]: pgmap v105: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:58:33.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:33 smithi184 ceph-mon[102844]: pgmap v105: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:58:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:33 smithi196 ceph-mon[76349]: pgmap v105: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T19:58:33.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 podman[122546]: 2024-09-20 19:58:33.728405199 +0000 UTC m=+0.556793891 container remove 61e416b9095eff383921d821b21f9f7885a8bf3ec702c6fd224f54dddff63043 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T19:58:33.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T19:58:33.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 systemd[1]: Stopped Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:58:33.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.1.service: Consumed 1.792s CPU time. 2024-09-20T19:58:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:33.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:33.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:34.169 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:33 smithi184 systemd[1]: Starting Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:58:34.483 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 podman[122644]: 2024-09-20 19:58:34.070929877 +0000 UTC m=+0.015907619 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:34.483 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 podman[122644]: 2024-09-20 19:58:34.170912009 +0000 UTC m=+0.115889745 container create 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:58:34.483 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 podman[122644]: 2024-09-20 19:58:34.382108638 +0000 UTC m=+0.327086383 container init 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:58:34.483 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 podman[122644]: 2024-09-20 19:58:34.385725423 +0000 UTC m=+0.330703161 container start 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-20T19:58:34.483 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 podman[122644]: 2024-09-20 19:58:34.481928465 +0000 UTC m=+0.426906215 container attach 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-20T19:58:34.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:34.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 bash[122644]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:34.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:34.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 bash[122644]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:58:35.265 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:35.266 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 bash[122644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:35.266 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:35.266 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:34 smithi184 bash[122644]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-mon[100622]: pgmap v107: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-mon[102844]: pgmap v107: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:35.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:35.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 bash[122644]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T19:58:35.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T19:58:35.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 bash[122644]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T19:58:35.549 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 bash[122644]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 bash[122644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate[122660]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 bash[122644]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T19:58:35.550 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:35 smithi184 podman[122644]: 2024-09-20 19:58:35.315309844 +0000 UTC m=+1.260287579 container died 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T19:58:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:35 smithi196 ceph-mon[76349]: pgmap v107: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T19:58:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:36 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:36.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:36 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:36.448 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 podman[122789]: 2024-09-20 19:58:36.102671966 +0000 UTC m=+0.781264926 container remove 1b56be416009a0dce84e10d78a3917ecb592f518c70be41595d6d9c44554132e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True) 2024-09-20T19:58:36.448 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 podman[122848]: 2024-09-20 19:58:36.2133953 +0000 UTC m=+0.016028997 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:58:36.448 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 podman[122848]: 2024-09-20 19:58:36.328622482 +0000 UTC m=+0.131256167 container create 1eab7b64494b99f93841cf53f1917dc39711d8869786679f4a8cc4efcbcbfc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2) 2024-09-20T19:58:36.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:36 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 90/636 objects degraded (14.151%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:36.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 podman[122848]: 2024-09-20 19:58:36.57323045 +0000 UTC m=+0.375864139 container init 1eab7b64494b99f93841cf53f1917dc39711d8869786679f4a8cc4efcbcbfc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-20T19:58:36.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 podman[122848]: 2024-09-20 19:58:36.576482326 +0000 UTC m=+0.379116021 container start 1eab7b64494b99f93841cf53f1917dc39711d8869786679f4a8cc4efcbcbfc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:58:36.798 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 ceph-osd[122867]: -- 172.21.15.184:0/570053391 <== mon.2 v2:172.21.15.184:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629712f5a40 con 0x5629720c4400 2024-09-20T19:58:36.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 bash[122848]: 1eab7b64494b99f93841cf53f1917dc39711d8869786679f4a8cc4efcbcbfc93 2024-09-20T19:58:36.799 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:36 smithi184 systemd[1]: Started Ceph osd.1 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: pgmap v108: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: pgmap v108: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: pgmap v108: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:37.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.547 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[122863]: 2024-09-20T19:58:38.473+0000 7febf91f9640 -1 Falling back to public interface 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: pgmap v109: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: pgmap v109: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: pgmap v109: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:39.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:39.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:39.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:40.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: pgmap v110: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: pgmap v110: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: pgmap v110: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:41.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:41.923 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[122863]: 2024-09-20T19:58:41.811+0000 7febf91f9640 -1 osd.1 121 log_to_monitors true 2024-09-20T19:58:42.271 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:42] "GET /metrics HTTP/1.1" 200 38794 "" "Prometheus/2.43.0" 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:42.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.048 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 20 19:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-1[122863]: 2024-09-20T19:58:42.775+0000 7febf0fa3640 -1 osd.1 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:58:43.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: pgmap v111: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: pgmap v111: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:43.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:43.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: pgmap v111: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:43.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:43.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:43.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:43.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:43.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:43.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:43.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:44.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430] boot 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430] boot 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: osd.1 [v2:172.21.15.184:6810/1425318430,v1:172.21.15.184:6811/1425318430] boot 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:44.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:44.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: pgmap v113: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: pgmap v113: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.365 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.366 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: pgmap v113: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 90/636 objects degraded (14.151%) 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:45.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:45.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:45.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:45.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: pgmap v116: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: pgmap v116: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: pgmap v116: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:46.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.353 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.354 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T19:58:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:47.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.504 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:58:48.243+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: pgmap v117: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T19:58:48.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: pgmap v117: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:58:48.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:48 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: pgmap v117: 161 pgs: 38 peering, 12 active+undersized, 7 active+undersized+degraded, 104 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/636 objects degraded (4.874%) 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:58:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:58:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:48 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T19:58:49.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:49 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:49.910 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:49 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:49 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/636 objects degraded (4.874%), 7 pgs degraded) 2024-09-20T19:58:51.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:50 smithi184 ceph-mon[100622]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:58:51.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:50 smithi184 ceph-mon[102844]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:58:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:50 smithi196 ceph-mon[76349]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T19:58:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:58:52] "GET /metrics HTTP/1.1" 200 38656 "" "Prometheus/2.43.0" 2024-09-20T19:58:52.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:52 smithi184 ceph-mon[100622]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:58:52.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:52 smithi184 ceph-mon[102844]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:58:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:52 smithi196 ceph-mon[76349]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:58:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:53.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:53.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:55.690 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:58:56.222 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 15m - - 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 15m - - 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (9m) 1s ago 10m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (10m) 18s ago 10m 89.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3m) 1s ago 8m 62.3M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3m) 18s ago 14m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4m) 1s ago 16m 509M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2m) 1s ago 16m 57.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2m) 18s ago 15m 45.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2m) 1s ago 15m 55.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (10m) 1s ago 10m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (10m) 18s ago 10m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:58:56.223 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (72s) 1s ago 14m 43.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (20s) 1s ago 14m 41.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (13m) 1s ago 13m 60.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 17b895a02b0d 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (13m) 1s ago 13m 55.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (12m) 18s ago 12m 68.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (12m) 18s ago 12m 58.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (12m) 18s ago 12m 53.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (11m) 18s ago 11m 56.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4m) 18s ago 11m 53.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (9m) 1s ago 9m 80.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (9m) 18s ago 9m 81.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (9m) 1s ago 9m 85.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:58:56.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (9m) 18s ago 8m 83.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:58:56.532 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:58:56.532 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:58:56.532 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:58:56.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:58:56.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 10, 2024-09-20T19:58:56.534 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T19:58:56.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:58:56.534 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "7/23 daemons upgraded", 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:58:56.801 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:58:57.113 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:57.113 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/222488846' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1360516663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/222488846' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1360516663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.44316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/222488846' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:58:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1360516663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:58:58.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:58 smithi196 ceph-mon[76349]: from='client.54296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:58.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[100622]: from='client.54296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[102844]: from='client.54296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:58:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:58.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T19:58:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:58:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:58:58.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:58:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:59 smithi196 ceph-mon[76349]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:58:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[100622]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[102844]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:58:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:58:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:58:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:01 smithi184 ceph-mon[100622]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:01 smithi184 ceph-mon[102844]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:01 smithi196 ceph-mon[76349]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:02] "GET /metrics HTTP/1.1" 200 38795 "" "Prometheus/2.43.0" 2024-09-20T19:59:03.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:03 smithi184 ceph-mon[100622]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:03.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:03 smithi184 ceph-mon[102844]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:03 smithi196 ceph-mon[76349]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:03.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:03.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:59:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:59:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[100622]: Upgrade: osd.2 is safe to restart 2024-09-20T19:59:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:59:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:59:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:04 smithi184 ceph-mon[102844]: Upgrade: osd.2 is safe to restart 2024-09-20T19:59:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T19:59:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:04 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T19:59:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:04 smithi196 ceph-mon[76349]: Upgrade: osd.2 is safe to restart 2024-09-20T19:59:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[100622]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[102844]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:59:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:05 smithi196 ceph-mon[76349]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T19:59:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:06 smithi184 ceph-mon[100622]: Upgrade: Updating osd.2 2024-09-20T19:59:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:06 smithi184 ceph-mon[100622]: Deploying daemon osd.2 on smithi184 2024-09-20T19:59:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:06 smithi184 ceph-mon[102844]: Upgrade: Updating osd.2 2024-09-20T19:59:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:06 smithi184 ceph-mon[102844]: Deploying daemon osd.2 on smithi184 2024-09-20T19:59:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:06 smithi196 ceph-mon[76349]: Upgrade: Updating osd.2 2024-09-20T19:59:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:06 smithi196 ceph-mon[76349]: Deploying daemon osd.2 on smithi184 2024-09-20T19:59:07.329 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:07.329 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-mon[100622]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:07.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:07.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-mon[102844]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:07 smithi196 ceph-mon[76349]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T19:59:07.959 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:07 smithi184 systemd[1]: Stopping Ceph osd.2 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:59:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:08 smithi184 ceph-mon[100622]: osd.2 marked itself down and dead 2024-09-20T19:59:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:08 smithi184 ceph-mon[102844]: osd.2 marked itself down and dead 2024-09-20T19:59:08.298 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[41348]: 2024-09-20T19:59:07.957+0000 7fd25ad66640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:59:08.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[41348]: 2024-09-20T19:59:07.957+0000 7fd25ad66640 -1 osd.2 126 *** Got signal Terminated *** 2024-09-20T19:59:08.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:07 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[41348]: 2024-09-20T19:59:07.957+0000 7fd25ad66640 -1 osd.2 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:59:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:08 smithi196 ceph-mon[76349]: osd.2 marked itself down and dead 2024-09-20T19:59:08.660 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:08 smithi184 podman[127167]: 2024-09-20 19:59:08.519147366 +0000 UTC m=+0.664777055 container died 17b895a02b0d7821322e22c5be169bf241059f1a5b20819dc335852600f82550 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T19:59:08.966 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:08.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T19:59:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[100622]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[100622]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T19:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[102844]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:09 smithi184 ceph-mon[102844]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T19:59:09.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:08 smithi184 podman[127167]: 2024-09-20 19:59:08.965725356 +0000 UTC m=+1.111355034 container cleanup 17b895a02b0d7821322e22c5be169bf241059f1a5b20819dc335852600f82550 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T19:59:09.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:08 smithi184 bash[127167]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2 2024-09-20T19:59:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:09 smithi196 ceph-mon[76349]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T19:59:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:09 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:09 smithi196 ceph-mon[76349]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T19:59:09.729 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:09 smithi184 podman[127264]: 2024-09-20 19:59:09.37667274 +0000 UTC m=+0.859297590 container remove 17b895a02b0d7821322e22c5be169bf241059f1a5b20819dc335852600f82550 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T19:59:10.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:10 smithi184 ceph-mon[100622]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T19:59:10.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:10 smithi184 ceph-mon[102844]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T19:59:10.495 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.287448373 +0000 UTC m=+0.015947836 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:10.495 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.389362785 +0000 UTC m=+0.117862277 container create a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T19:59:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:10 smithi196 ceph-mon[76349]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T19:59:10.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.608383906 +0000 UTC m=+0.336883363 container init a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T19:59:10.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.611827234 +0000 UTC m=+0.340326685 container start a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-20T19:59:10.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.70854997 +0000 UTC m=+0.437049421 container attach a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True) 2024-09-20T19:59:11.113 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:10 smithi184 podman[127473]: 2024-09-20 19:59:10.80330442 +0000 UTC m=+0.531803884 container died a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-20T19:59:11.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:11 smithi184 ceph-mon[100622]: pgmap v130: 161 pgs: 22 active+undersized, 4 stale+active+clean, 11 active+undersized+degraded, 124 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 32/636 objects degraded (5.031%) 2024-09-20T19:59:11.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:11 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 32/636 objects degraded (5.031%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:11.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:11 smithi184 ceph-mon[102844]: pgmap v130: 161 pgs: 22 active+undersized, 4 stale+active+clean, 11 active+undersized+degraded, 124 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 32/636 objects degraded (5.031%) 2024-09-20T19:59:11.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:11 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 32/636 objects degraded (5.031%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:11 smithi196 ceph-mon[76349]: pgmap v130: 161 pgs: 22 active+undersized, 4 stale+active+clean, 11 active+undersized+degraded, 124 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 32/636 objects degraded (5.031%) 2024-09-20T19:59:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:11 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 32/636 objects degraded (5.031%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:11.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:11 smithi184 podman[127510]: 2024-09-20 19:59:11.504497129 +0000 UTC m=+0.693857115 container remove a4e7c86ea7422255931df05a4cb0428705b0b1d1c76bd897df1abceb22c2e2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T19:59:11.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:11 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-20T19:59:11.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:11 smithi184 systemd[1]: Stopped Ceph osd.2 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:59:11.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:11 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.2.service: Consumed 6.699s CPU time. 2024-09-20T19:59:11.798 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:11 smithi184 systemd[1]: Starting Ceph osd.2 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:59:12.124 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:12] "GET /metrics HTTP/1.1" 200 38795 "" "Prometheus/2.43.0" 2024-09-20T19:59:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-mon[100622]: pgmap v131: 161 pgs: 28 active+undersized, 2 stale+active+clean, 12 active+undersized+degraded, 119 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:59:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-mon[102844]: pgmap v131: 161 pgs: 28 active+undersized, 2 stale+active+clean, 12 active+undersized+degraded, 119 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:59:13.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 podman[127720]: 2024-09-20 19:59:13.122125612 +0000 UTC m=+0.094844497 container create 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T19:59:13.299 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 podman[127720]: 2024-09-20 19:59:13.048459286 +0000 UTC m=+0.021178173 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:13 smithi196 ceph-mon[76349]: pgmap v131: 161 pgs: 28 active+undersized, 2 stale+active+clean, 12 active+undersized+degraded, 119 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 37/636 objects degraded (5.818%) 2024-09-20T19:59:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:13.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:13.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 podman[127720]: 2024-09-20 19:59:13.357742409 +0000 UTC m=+0.330461291 container init 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 podman[127720]: 2024-09-20 19:59:13.361497165 +0000 UTC m=+0.334216040 container start 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 podman[127720]: 2024-09-20 19:59:13.42448987 +0000 UTC m=+0.397208760 container attach 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 bash[127720]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:13.799 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 bash[127720]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:14.289 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T19:59:14.289 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:13 smithi184 bash[127720]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T19:59:14.289 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T19:59:14.289 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T19:59:14.289 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T19:59:14.548 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T19:59:14.548 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T19:59:14.548 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T19:59:14.548 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T19:59:14.548 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T19:59:14.549 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T19:59:14.549 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T19:59:14.549 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate[127738]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T19:59:14.549 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 bash[127720]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T19:59:14.549 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:14 smithi184 podman[127720]: 2024-09-20 19:59:14.321604365 +0000 UTC m=+1.294323251 container died 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:59:14.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:14.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: pgmap v132: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 56/636 objects degraded (8.805%) 2024-09-20T19:59:14.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.918 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.918 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:14.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: pgmap v132: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 56/636 objects degraded (8.805%) 2024-09-20T19:59:14.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: pgmap v132: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 56/636 objects degraded (8.805%) 2024-09-20T19:59:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:15.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:15 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 56/636 objects degraded (8.805%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:15.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:15 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 56/636 objects degraded (8.805%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:15.940 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:15 smithi184 podman[128000]: 2024-09-20 19:59:15.662172018 +0000 UTC m=+1.333598367 container remove 2d93a3f403606b127f0cb25dfeda6d2a8c991e0f02fa5c117fcaf85325530ddb (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T19:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:15 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 56/636 objects degraded (8.805%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:16.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:16.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:16 smithi184 ceph-mon[100622]: pgmap v133: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:16.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:16.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:16 smithi184 ceph-mon[102844]: pgmap v133: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:16.764 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 podman[128162]: 2024-09-20 19:59:16.49181275 +0000 UTC m=+0.736069644 container create 04c43acf3cece9ffeda9f2a4857e263ac7735921814f010ec4bb5393fffff4ac (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T19:59:16.764 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 podman[128162]: 2024-09-20 19:59:16.401423556 +0000 UTC m=+0.645680460 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:16.764 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 podman[128162]: 2024-09-20 19:59:16.737203231 +0000 UTC m=+0.981460122 container init 04c43acf3cece9ffeda9f2a4857e263ac7735921814f010ec4bb5393fffff4ac (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T19:59:16.764 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 podman[128162]: 2024-09-20 19:59:16.7406055 +0000 UTC m=+0.984862386 container start 04c43acf3cece9ffeda9f2a4857e263ac7735921814f010ec4bb5393fffff4ac (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:59:16.764 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 ceph-osd[128242]: -- 172.21.15.184:0/2865932404 <== mon.2 v2:172.21.15.184:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610183c5a40 con 0x5610183a3c00 2024-09-20T19:59:17.048 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 bash[128162]: 04c43acf3cece9ffeda9f2a4857e263ac7735921814f010ec4bb5393fffff4ac 2024-09-20T19:59:17.048 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:16 smithi184 systemd[1]: Started Ceph osd.2 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:59:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:16 smithi196 ceph-mon[76349]: pgmap v133: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:18.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T19:59:17.735+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.054 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.056 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.057 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:18.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:18.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:19.043 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[128238]: 2024-09-20T19:59:18.669+0000 7f0d959f3640 -1 Falling back to public interface 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: pgmap v134: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: pgmap v135: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 227 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: pgmap v136: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[100622]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: pgmap v134: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: pgmap v135: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 227 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: pgmap v136: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:19 smithi184 ceph-mon[102844]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:59:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: pgmap v134: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 214 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: pgmap v135: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 227 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: pgmap v136: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:19 smithi196 ceph-mon[76349]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T19:59:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:21 smithi184 ceph-mon[100622]: pgmap v137: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:21 smithi184 ceph-mon[102844]: pgmap v137: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:21 smithi196 ceph-mon[76349]: pgmap v137: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:22] "GET /metrics HTTP/1.1" 200 38791 "" "Prometheus/2.43.0" 2024-09-20T19:59:22.298 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[128238]: 2024-09-20T19:59:22.207+0000 7f0d959f3640 -1 osd.2 126 log_to_monitors true 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[100622]: pgmap v138: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[100622]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[100622]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[102844]: pgmap v138: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[102844]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-mon[102844]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.534 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-2[128238]: 2024-09-20T19:59:23.250+0000 7f0d89795640 -1 osd.2 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:59:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:23 smithi196 ceph-mon[76349]: pgmap v138: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:23 smithi196 ceph-mon[76349]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:23 smithi196 ceph-mon[76349]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T19:59:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:23.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:23.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[100622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:59:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[100622]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T19:59:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[100622]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[100622]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[102844]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:59:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[102844]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T19:59:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[102844]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:24 smithi184 ceph-mon[102844]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:24 smithi196 ceph-mon[76349]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:59:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:24 smithi196 ceph-mon[76349]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T19:59:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:24 smithi196 ceph-mon[76349]: from='osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:24 smithi196 ceph-mon[76349]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: pgmap v140: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507] boot 2024-09-20T19:59:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[100622]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: pgmap v140: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507] boot 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:25 smithi184 ceph-mon[102844]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: pgmap v140: 161 pgs: 32 active+undersized, 14 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: osd.2 [v2:172.21.15.184:6818/481828507,v1:172.21.15.184:6819/481828507] boot 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:25 smithi196 ceph-mon[76349]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T19:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[100622]: pgmap v142: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[100622]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state']) 2024-09-20T19:59:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[102844]: pgmap v142: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:26 smithi184 ceph-mon[102844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state']) 2024-09-20T19:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:26 smithi196 ceph-mon[76349]: pgmap v142: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:26 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:26 smithi196 ceph-mon[76349]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi184 is in unknown state']) 2024-09-20T19:59:27.410 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:59:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:27.940 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 16m - - 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 15m - - 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (10m) 2s ago 11m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (11m) 9s ago 11m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4m) 2s ago 9m 62.3M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4m) 9s ago 15m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5m) 2s ago 17m 510M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3m) 2s ago 17m 58.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2m) 9s ago 15m 46.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3m) 2s ago 15m 55.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (11m) 2s ago 11m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (10m) 9s ago 10m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:59:27.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (104s) 2s ago 15m 44.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (51s) 2s ago 14m 42.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (11s) 2s ago 14m 12.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (13m) 2s ago 13m 56.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 94cadaf1931a 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (13m) 9s ago 13m 68.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (13m) 9s ago 13m 58.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (12m) 9s ago 12m 54.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (12m) 9s ago 12m 58.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4m) 9s ago 11m 54.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (9m) 2s ago 9m 80.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (10m) 9s ago 10m 82.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (9m) 2s ago 9m 85.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:59:27.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (9m) 9s ago 9m 84.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T19:59:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:59:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:59:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T19:59:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T19:59:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 9, 2024-09-20T19:59:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T19:59:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T19:59:28.260 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:59:28.527 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "8/23 daemons upgraded", 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T19:59:28.528 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T19:59:28.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:28 smithi184 ceph-mon[100622]: pgmap v144: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:28.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:28 smithi184 ceph-mon[102844]: pgmap v144: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:28 smithi196 ceph-mon[76349]: pgmap v144: 161 pgs: 29 peering, 14 active+undersized, 3 active+undersized+degraded, 115 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/636 objects degraded (3.145%) 2024-09-20T19:59:28.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:28.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T19:59:28.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:28.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:28.842 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-09-20T19:59:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='client.54326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4137602058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4148046299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='client.54326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4137602058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4148046299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='client.44346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='client.54326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4137602058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4148046299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T19:59:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:30 smithi196 ceph-mon[76349]: from='client.54338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:30 smithi196 ceph-mon[76349]: pgmap v145: 161 pgs: 29 peering, 132 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:59:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:30 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded) 2024-09-20T19:59:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[100622]: from='client.54338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[100622]: pgmap v145: 161 pgs: 29 peering, 132 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:59:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded) 2024-09-20T19:59:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[102844]: from='client.54338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:59:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[102844]: pgmap v145: 161 pgs: 29 peering, 132 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T19:59:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:30 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/636 objects degraded (3.145%), 3 pgs degraded) 2024-09-20T19:59:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:32 smithi184 ceph-mon[100622]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T19:59:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:32 smithi184 ceph-mon[102844]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T19:59:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:32] "GET /metrics HTTP/1.1" 200 38795 "" "Prometheus/2.43.0" 2024-09-20T19:59:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:32 smithi196 ceph-mon[76349]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T19:59:33.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:33.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T19:59:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:33.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:33.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: Upgrade: osd.3 is safe to restart 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: Upgrade: osd.3 is safe to restart 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:59:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T19:59:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: Upgrade: osd.3 is safe to restart 2024-09-20T19:59:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T19:59:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T19:59:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:35 smithi184 ceph-mon[100622]: Upgrade: Updating osd.3 2024-09-20T19:59:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:35 smithi184 ceph-mon[100622]: Deploying daemon osd.3 on smithi184 2024-09-20T19:59:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:35 smithi184 ceph-mon[102844]: Upgrade: Updating osd.3 2024-09-20T19:59:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:35 smithi184 ceph-mon[102844]: Deploying daemon osd.3 on smithi184 2024-09-20T19:59:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:35 smithi196 ceph-mon[76349]: Upgrade: Updating osd.3 2024-09-20T19:59:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:35 smithi196 ceph-mon[76349]: Deploying daemon osd.3 on smithi184 2024-09-20T19:59:36.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:36 smithi184 ceph-mon[100622]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T19:59:36.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:36 smithi184 ceph-mon[102844]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T19:59:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:36 smithi196 ceph-mon[76349]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T19:59:37.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:37.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:37.429 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:37 smithi184 systemd[1]: Stopping Ceph osd.3 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:59:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:37.798 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[44924]: 2024-09-20T19:59:37.426+0000 7f189d903640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:59:37.798 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[44924]: 2024-09-20T19:59:37.426+0000 7f189d903640 -1 osd.3 131 *** Got signal Terminated *** 2024-09-20T19:59:37.798 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:37 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[44924]: 2024-09-20T19:59:37.426+0000 7f189d903640 -1 osd.3 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T19:59:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:38 smithi184 ceph-mon[100622]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T19:59:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:38 smithi184 ceph-mon[100622]: osd.3 marked itself down and dead 2024-09-20T19:59:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:38 smithi184 ceph-mon[102844]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T19:59:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:38 smithi184 ceph-mon[102844]: osd.3 marked itself down and dead 2024-09-20T19:59:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:38 smithi196 ceph-mon[76349]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T19:59:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:38 smithi196 ceph-mon[76349]: osd.3 marked itself down and dead 2024-09-20T19:59:38.842 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:38.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:39.158 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:38 smithi184 podman[130179]: 2024-09-20 19:59:38.841119633 +0000 UTC m=+1.537071818 container died 94cadaf1931ad3566430cb5e5e910a804dc87538e353dbe95a10ff4c02707fae (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T19:59:39.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:39.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.476 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:39 smithi184 podman[130179]: 2024-09-20 19:59:39.307038067 +0000 UTC m=+2.002990249 container cleanup 94cadaf1931ad3566430cb5e5e910a804dc87538e353dbe95a10ff4c02707fae (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T19:59:39.476 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:39 smithi184 bash[130179]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T19:59:39.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T19:59:39.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T19:59:39.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.632 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:39.798 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:39 smithi184 podman[130225]: 2024-09-20 19:59:39.474530853 +0000 UTC m=+0.758904805 container remove 94cadaf1931ad3566430cb5e5e910a804dc87538e353dbe95a10ff4c02707fae (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d) 2024-09-20T19:59:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:40 smithi196 ceph-mon[76349]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T19:59:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:40 smithi196 ceph-mon[76349]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T19:59:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:40 smithi184 ceph-mon[100622]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T19:59:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:40 smithi184 ceph-mon[100622]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T19:59:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:40 smithi184 ceph-mon[102844]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T19:59:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:40 smithi184 ceph-mon[102844]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T19:59:40.549 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.18008268 +0000 UTC m=+0.018775997 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:40.549 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.302952339 +0000 UTC m=+0.141645645 container create f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:59:41.048 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.560326713 +0000 UTC m=+0.399020027 container init f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-20T19:59:41.048 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.563818653 +0000 UTC m=+0.402511975 container start f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T19:59:41.048 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.664044124 +0000 UTC m=+0.502737432 container attach f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T19:59:41.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:40 smithi184 podman[130357]: 2024-09-20 19:59:40.764743822 +0000 UTC m=+0.603437152 container died f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T19:59:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:41 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 62/636 objects degraded (9.748%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:41.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:41 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 62/636 objects degraded (9.748%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:41.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:41 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 62/636 objects degraded (9.748%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:41.702 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:41 smithi184 podman[130477]: 2024-09-20 19:59:41.452150196 +0000 UTC m=+0.681171442 container remove f3918d734d1e4e9f897ce959b7881d590018a8936b9171a44b745cf2243566df (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T19:59:41.702 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:41 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-20T19:59:41.702 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:41 smithi184 systemd[1]: Stopped Ceph osd.3 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:59:41.703 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:41 smithi184 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.3.service: Consumed 8.710s CPU time. 2024-09-20T19:59:41.703 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:41 smithi184 systemd[1]: Starting Ceph osd.3 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T19:59:42.260 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:42] "GET /metrics HTTP/1.1" 200 38795 "" "Prometheus/2.43.0" 2024-09-20T19:59:42.260 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 podman[130624]: 2024-09-20 19:59:42.248436562 +0000 UTC m=+0.104365673 container create 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2) 2024-09-20T19:59:42.260 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 podman[130624]: 2024-09-20 19:59:42.161194606 +0000 UTC m=+0.017123672 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:42 smithi184 ceph-mon[100622]: pgmap v153: 161 pgs: 27 active+undersized, 7 stale+active+clean, 20 active+undersized+degraded, 107 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 31 op/s; 62/636 objects degraded (9.748%) 2024-09-20T19:59:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:42 smithi184 ceph-mon[102844]: pgmap v153: 161 pgs: 27 active+undersized, 7 stale+active+clean, 20 active+undersized+degraded, 107 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 31 op/s; 62/636 objects degraded (9.748%) 2024-09-20T19:59:42.548 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 podman[130624]: 2024-09-20 19:59:42.501717111 +0000 UTC m=+0.357646186 container init 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T19:59:42.548 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 podman[130624]: 2024-09-20 19:59:42.507783102 +0000 UTC m=+0.363712170 container start 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T19:59:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:42 smithi196 ceph-mon[76349]: pgmap v153: 161 pgs: 27 active+undersized, 7 stale+active+clean, 20 active+undersized+degraded, 107 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 31 op/s; 62/636 objects degraded (9.748%) 2024-09-20T19:59:43.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 podman[130624]: 2024-09-20 19:59:42.609779808 +0000 UTC m=+0.465708884 container attach 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T19:59:43.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:43.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 bash[130624]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:43.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:43.049 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:42 smithi184 bash[130624]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T19:59:43.438 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:59:43.438 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:59:43.438 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T19:59:43.438 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T19:59:43.784 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:43.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:43.784 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:43.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate[130656]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 bash[130624]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T19:59:43.785 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:43 smithi184 podman[130624]: 2024-09-20 19:59:43.582067757 +0000 UTC m=+1.437996830 container died 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T19:59:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[102844]: pgmap v154: 161 pgs: 33 active+undersized, 2 stale+active+clean, 25 active+undersized+degraded, 101 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 80/636 objects degraded (12.579%) 2024-09-20T19:59:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[100622]: pgmap v154: 161 pgs: 33 active+undersized, 2 stale+active+clean, 25 active+undersized+degraded, 101 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 80/636 objects degraded (12.579%) 2024-09-20T19:59:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:44 smithi196 ceph-mon[76349]: pgmap v154: 161 pgs: 33 active+undersized, 2 stale+active+clean, 25 active+undersized+degraded, 101 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 80/636 objects degraded (12.579%) 2024-09-20T19:59:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:44.845 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:44 smithi184 podman[130989]: 2024-09-20 19:59:44.576259707 +0000 UTC m=+1.091736891 container remove 5abe612f509afb79f6523aacbcf23d27ff1a44db4fb750ffc56ac89420c84c33 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[100622]: pgmap v155: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[102844]: pgmap v155: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.431 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:46 smithi184 podman[131100]: 2024-09-20 19:59:46.014268903 +0000 UTC m=+1.346561550 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T19:59:46.431 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:46 smithi184 podman[131100]: 2024-09-20 19:59:46.150923544 +0000 UTC m=+1.483216253 container create b06017b98bafbc20aeea44a9dfdad7332a20677801017288934b7e88b10567b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0) 2024-09-20T19:59:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:46 smithi196 ceph-mon[76349]: pgmap v155: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.283 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:47 smithi184 podman[131100]: 2024-09-20 19:59:47.015684067 +0000 UTC m=+2.347976727 container init b06017b98bafbc20aeea44a9dfdad7332a20677801017288934b7e88b10567b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T19:59:47.283 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:47 smithi184 podman[131100]: 2024-09-20 19:59:47.019013549 +0000 UTC m=+2.351306196 container start b06017b98bafbc20aeea44a9dfdad7332a20677801017288934b7e88b10567b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T19:59:47.284 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-osd[131323]: -- 172.21.15.184:0/3483731130 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56529d703a40 con 0x56529d6e1c00 2024-09-20T19:59:47.284 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:47 smithi184 bash[131100]: b06017b98bafbc20aeea44a9dfdad7332a20677801017288934b7e88b10567b4 2024-09-20T19:59:47.284 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:47 smithi184 systemd[1]: Started Ceph osd.3 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: pgmap v156: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: pgmap v157: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: pgmap v156: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: pgmap v157: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: pgmap v156: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: pgmap v157: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:48.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.098 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[131275]: 2024-09-20T19:59:48.915+0000 7f074387c640 -1 Falling back to public interface 2024-09-20T19:59:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:49.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: pgmap v158: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: pgmap v158: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.337 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.338 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.344 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.345 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: pgmap v158: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 81/636 objects degraded (12.736%) 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:50.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:51.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:19:59:52] "GET /metrics HTTP/1.1" 200 38792 "" "Prometheus/2.43.0" 2024-09-20T19:59:52.548 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[131275]: 2024-09-20T19:59:52.496+0000 7f074387c640 -1 osd.3 131 log_to_monitors true 2024-09-20T19:59:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: pgmap v159: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: pgmap v159: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: pgmap v159: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:52.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:52.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:52.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:52.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.637 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:53.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:53.638 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:53.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.651 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:53.652 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:53.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:53.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:53.653 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.298 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 20 19:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-3[131275]: 2024-09-20T19:59:53.942+0000 7f073761e640 -1 osd.3 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: pgmap v160: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: pgmap v160: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: pgmap v160: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913]' entity='osd.3' 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:54.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:54.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:54.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:54.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:55.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913] boot 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:55.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913] boot 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: osd.3 [v2:172.21.15.184:6826/3511736913,v1:172.21.15.184:6827/3511736913] boot 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T19:59:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:55.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:55.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:55.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:55.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: pgmap v163: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:59:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: pgmap v163: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: pgmap v163: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:56.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:56.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:56.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.567 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.567 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.572 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.574 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.575 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T19:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:57.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:57.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:57.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:57.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: pgmap v165: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: pgmap v165: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: pgmap v165: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 81/636 objects degraded (12.736%) 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:58.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:58.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T19:59:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 19:59:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T19:59:58.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T19:59:59.159 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T19:59:59.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 19:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T19:59:59.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 19:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T19:59:59.792 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:59:59.792 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 16m - - 2024-09-20T19:59:59.792 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 16m - - 2024-09-20T19:59:59.792 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (10m) 12s ago 11m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (11m) 1s ago 11m 91.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4m) 12s ago 9m 62.3M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4m) 1s ago 15m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5m) 12s ago 17m 511M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3m) 12s ago 17m 58.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3m) 1s ago 16m 48.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3m) 12s ago 16m 55.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (11m) 12s ago 11m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (11m) 1s ago 11m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2m) 12s ago 15m 46.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (83s) 12s ago 15m 44.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (43s) 12s ago 14m 44.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 starting - - - 4096M 2024-09-20T19:59:59.793 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (13m) 1s ago 13m 71.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 30744b286a89 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (13m) 1s ago 13m 60.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (13m) 1s ago 13m 56.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (12m) 1s ago 12m 61.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5m) 1s ago 12m 52.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (10m) 12s ago 10m 81.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (10m) 1s ago 10m 82.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (10m) 12s ago 10m 85.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T19:59:59.794 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (10m) 1s ago 10m 84.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:00:00.223 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4, 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:00:00.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:00.225 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:00:00.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8, 2024-09-20T20:00:00.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 9 2024-09-20T20:00:00.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:00:00.225 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:00:00.325 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.326 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:00:00.149+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-20T20:00:00.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pgmap v166: 161 pgs: 15 active+undersized, 12 active+undersized+degraded, 134 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 36/636 objects degraded (5.660%); 0 B/s, 4 objects/s recovering 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: [WRN] PG_DEGRADED: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.a is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.e is active+undersized+degraded, acting [2,7] 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-09-20T20:00:00.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.c is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-09-20T20:00:00.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 5.1 is active+undersized+degraded, acting [4,7] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:00.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4203560451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "9/23 daemons upgraded", 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:00:00.535 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:00:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pgmap v166: 161 pgs: 15 active+undersized, 12 active+undersized+degraded, 134 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 36/636 objects degraded (5.660%); 0 B/s, 4 objects/s recovering 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pgmap v166: 161 pgs: 15 active+undersized, 12 active+undersized+degraded, 134 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 36/636 objects degraded (5.660%); 0 B/s, 4 objects/s recovering 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: [WRN] PG_DEGRADED: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.a is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.e is active+undersized+degraded, acting [2,7] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: [WRN] PG_DEGRADED: Degraded data redundancy: 81/636 objects degraded (12.736%), 26 pgs degraded 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.a is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.e is active+undersized+degraded, acting [2,7] 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.c is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-09-20T20:00:00.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 5.1 is active+undersized+degraded, acting [4,7] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4203560451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.c is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-09-20T20:00:00.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 5.1 is active+undersized+degraded, acting [4,7] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:00.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4203560451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:00.861 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-09-20T20:00:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-09-20T20:00:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-09-20T20:00:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-09-20T20:00:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1 is active+undersized+degraded, acting [4,7] 2024-09-20T20:00:00.862 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='client.64264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='client.54368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1794623804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='client.64264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='client.54368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1794623804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:01.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='client.64264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='client.54368 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:01.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:01.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-20T20:00:01.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:01.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1794623804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:02] "GET /metrics HTTP/1.1" 200 38797 "" "Prometheus/2.43.0" 2024-09-20T20:00:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:02 smithi196 ceph-mon[76349]: from='client.54380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:02 smithi196 ceph-mon[76349]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:00:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:02 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded) 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[100622]: from='client.54380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[100622]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded) 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[102844]: from='client.54380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[102844]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 184 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:00:02.690 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:02 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/636 objects degraded (5.660%), 12 pgs degraded) 2024-09-20T20:00:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:03.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:03.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:04.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:04 smithi184 ceph-mon[100622]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 99 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T20:00:04.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:04 smithi184 ceph-mon[102844]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 99 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T20:00:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:04 smithi196 ceph-mon[76349]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 99 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T20:00:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:06 smithi196 ceph-mon[76349]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:06 smithi184 ceph-mon[100622]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:06 smithi184 ceph-mon[102844]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:08 smithi196 ceph-mon[76349]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:08.665 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:08 smithi184 ceph-mon[100622]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:08.665 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:08 smithi184 ceph-mon[102844]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:00:09.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:08 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:08.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:09 smithi184 ceph-mon[102844]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 69 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:00:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:09 smithi184 ceph-mon[100622]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 69 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:00:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:09 smithi196 ceph-mon[76349]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 69 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:00:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:12 smithi184 ceph-mon[100622]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T20:00:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:12 smithi184 ceph-mon[102844]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T20:00:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:12] "GET /metrics HTTP/1.1" 200 38797 "" "Prometheus/2.43.0" 2024-09-20T20:00:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:12 smithi196 ceph-mon[76349]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T20:00:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:13.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:13.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[102844]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[100622]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:14 smithi196 ceph-mon[76349]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:15.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[100622]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[100622]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[102844]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:16 smithi184 ceph-mon[102844]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:16.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:16 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:16.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:16 smithi196 ceph-mon[76349]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:16.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:16 smithi196 ceph-mon[76349]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:18 smithi196 ceph-mon[76349]: Upgrade: Updating osd.4 2024-09-20T20:00:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:18 smithi196 ceph-mon[76349]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:18 smithi196 ceph-mon[76349]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[100622]: Upgrade: Updating osd.4 2024-09-20T20:00:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[100622]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[100622]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[102844]: Upgrade: Updating osd.4 2024-09-20T20:00:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[102844]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:18 smithi184 ceph-mon[102844]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:19.670 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:19.670 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:19.670 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:19.671 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:19 smithi196 ceph-mon[76349]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:19.981 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:19 smithi196 systemd[1]: Stopping Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:00:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[100622]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:19 smithi184 ceph-mon[102844]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:20.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-mon[76349]: osd.4 marked itself down and dead 2024-09-20T20:00:20.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:20.772 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[31898]: 2024-09-20T20:00:20.459+0000 7f8965b2e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:00:20.772 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[31898]: 2024-09-20T20:00:20.459+0000 7f8965b2e640 -1 osd.4 136 *** Got signal Terminated *** 2024-09-20T20:00:20.773 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:20 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[31898]: 2024-09-20T20:00:20.459+0000 7f8965b2e640 -1 osd.4 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:00:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[100622]: osd.4 marked itself down and dead 2024-09-20T20:00:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[102844]: osd.4 marked itself down and dead 2024-09-20T20:00:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:21.316 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:21 smithi196 podman[91297]: 2024-09-20 20:00:21.033418377 +0000 UTC m=+0.710279908 container died 30744b286a89e3e3b34add50c4f0579dd73e5319ff33fc7bc55ffa74797a9345 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d) 2024-09-20T20:00:21.568 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:21 smithi196 podman[91297]: 2024-09-20 20:00:21.412908289 +0000 UTC m=+1.089769747 container cleanup 30744b286a89e3e3b34add50c4f0579dd73e5319ff33fc7bc55ffa74797a9345 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T20:00:21.569 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:21 smithi196 bash[91297]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4 2024-09-20T20:00:21.569 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:21 smithi196 podman[91316]: 2024-09-20 20:00:21.564371674 +0000 UTC m=+0.628797235 container remove 30744b286a89e3e3b34add50c4f0579dd73e5319ff33fc7bc55ffa74797a9345 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d) 2024-09-20T20:00:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:21 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:00:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:21 smithi196 ceph-mon[76349]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:00:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:21 smithi196 ceph-mon[76349]: pgmap v178: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[100622]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[100622]: pgmap v178: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[102844]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:00:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:21 smithi184 ceph-mon[102844]: pgmap v178: 161 pgs: 24 stale+active+clean, 137 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:22.293 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:21.924433986 +0000 UTC m=+0.022587746 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:00:22.294 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:22.040315808 +0000 UTC m=+0.138469556 container create e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:00:22.294 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:22.292710622 +0000 UTC m=+0.390864382 container init e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:00:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:22] "GET /metrics HTTP/1.1" 200 38789 "" "Prometheus/2.43.0" 2024-09-20T20:00:22.586 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:22.296149265 +0000 UTC m=+0.394303019 container start e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:00:22.586 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:22.367745442 +0000 UTC m=+0.465899194 container attach e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T20:00:22.586 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:22 smithi196 podman[91417]: 2024-09-20 20:00:22.487133649 +0000 UTC m=+0.585287399 container died e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T20:00:22.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:22 smithi184 ceph-mon[100622]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:00:22.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:22 smithi184 ceph-mon[102844]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:00:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:22 smithi196 ceph-mon[76349]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:00:23.592 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91442]: 2024-09-20 20:00:23.179895432 +0000 UTC m=+0.686879067 container remove e00633ba7d303b8e07e682cb83bd47b1916cdf1c6ace483307d97b23e5f81cc6 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T20:00:23.593 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-20T20:00:23.593 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 systemd[1]: Stopped Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:00:23.593 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.4.service: Consumed 9.653s CPU time. 2024-09-20T20:00:23.593 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 systemd[1]: Starting Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:00:23.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:23 smithi184 ceph-mon[100622]: pgmap v180: 161 pgs: 5 active+undersized, 21 stale+active+clean, 5 active+undersized+degraded, 130 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:00:23.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:23 smithi184 ceph-mon[102844]: pgmap v180: 161 pgs: 5 active+undersized, 21 stale+active+clean, 5 active+undersized+degraded, 130 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:00:23.705 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:23.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:23.705 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:23.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:23.843 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:23 smithi196 ceph-mon[76349]: pgmap v180: 161 pgs: 5 active+undersized, 21 stale+active+clean, 5 active+undersized+degraded, 130 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:00:23.843 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91544]: 2024-09-20 20:00:23.526712853 +0000 UTC m=+0.016058191 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:00:23.843 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91544]: 2024-09-20 20:00:23.648260133 +0000 UTC m=+0.137605503 container create 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91544]: 2024-09-20 20:00:23.842104506 +0000 UTC m=+0.331449847 container init 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91544]: 2024-09-20 20:00:23.84538332 +0000 UTC m=+0.334728649 container start 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 podman[91544]: 2024-09-20 20:00:23.917134444 +0000 UTC m=+0.406479780 container attach 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 bash[91544]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:00:24.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:23 smithi196 bash[91544]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:00:24.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:24.742 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:00:24.742 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:00:24.742 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:00:24.742 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:00:24.742 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:00:24.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:24 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:24.941 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:24 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:25.054 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:00:25.054 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:00:25.054 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:00:25.054 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[91559]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 bash[91544]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:00:25.055 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:24 smithi196 podman[91544]: 2024-09-20 20:00:24.771383565 +0000 UTC m=+1.260728902 container died 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-20T20:00:25.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:25 smithi196 ceph-mon[76349]: pgmap v181: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:25.711 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 podman[91686]: 2024-09-20 20:00:25.4291645 +0000 UTC m=+0.651730450 container remove 7f79aac777a241656ce2ae473833fd010ee14394042ed300337ef7c936a96e2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T20:00:25.711 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 podman[91744]: 2024-09-20 20:00:25.545448443 +0000 UTC m=+0.016499109 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:00:25.711 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 podman[91744]: 2024-09-20 20:00:25.710366426 +0000 UTC m=+0.181417072 container create 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:00:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:25 smithi184 ceph-mon[100622]: pgmap v181: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:25 smithi184 ceph-mon[102844]: pgmap v181: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:26.025 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 podman[91744]: 2024-09-20 20:00:25.932689383 +0000 UTC m=+0.403740042 container init 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:00:26.026 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 podman[91744]: 2024-09-20 20:00:25.935813004 +0000 UTC m=+0.406863652 container start 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T20:00:26.026 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:25 smithi196 ceph-osd[91762]: -- 172.21.15.196:0/4276170932 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610fd4f5a40 con 0x5610fe2c8400 2024-09-20T20:00:26.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:26 smithi196 bash[91744]: 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c 2024-09-20T20:00:26.276 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:26 smithi196 systemd[1]: Started Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:27.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:27.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:27.639 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:27.569+0000 7fd4b00b6640 -1 Falling back to public interface 2024-09-20T20:00:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: pgmap v182: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: pgmap v182: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: pgmap v182: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:28.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:28.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:28.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:29.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: pgmap v183: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: pgmap v183: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: pgmap v183: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:30.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:30.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:30.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:30.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.087 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:30.886+0000 7fd4b00b6640 -1 osd.4 136 log_to_monitors true 2024-09-20T20:00:31.178 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:31.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 17m - - 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 17m - - 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (11m) 11s ago 12m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (12m) 13s ago 12m 88.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5m) 11s ago 10m 62.4M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5m) 13s ago 16m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6m) 11s ago 18m 513M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:00:31.769 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4m) 11s ago 18m 61.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3m) 13s ago 16m 48.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4m) 11s ago 16m 58.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (12m) 11s ago 12m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (12m) 13s ago 12m 16.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2m) 11s ago 16m 47.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (115s) 11s ago 15m 46.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (75s) 11s ago 15m 46.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (45s) 11s ago 14m 52.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 starting - - - 4096M 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (14m) 13s ago 14m 61.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (13m) 13s ago 13m 57.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (13m) 13s ago 13m 63.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:00:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5m) 13s ago 12m 53.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:00:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (11m) 11s ago 11m 81.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:00:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (11m) 13s ago 11m 82.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:00:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (10m) 11s ago 10m 86.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:00:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (10m) 13s ago 10m 84.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:00:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:32] "GET /metrics HTTP/1.1" 200 38790 "" "Prometheus/2.43.0" 2024-09-20T20:00:32.091 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:31 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:31.857+0000 7fd4a7e60640 -1 osd.4 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:00:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3, 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 9 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:00:32.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:00:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='client.54395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: pgmap v184: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: osdmap e139: 8 total, 7 up, 8 in 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.412 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:00:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:00:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "9/23 daemons upgraded", 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:00:32.413 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:00:32.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='client.54395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: pgmap v184: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='client.54395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: pgmap v184: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 272 MiB used, 715 GiB / 715 GiB avail; 108/636 objects degraded (16.981%) 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: osdmap e139: 8 total, 7 up, 8 in 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: osdmap e139: 8 total, 7 up, 8 in 2024-09-20T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: osd.4 (root=default,host=smithi196) is down 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-09-20T20:00:32.757 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-09-20T20:00:32.758 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.d is active+undersized+degraded, acting [2,1] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.e is active+undersized+degraded, acting [6,1] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1 is active+undersized+degraded, acting [3,7] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1d is active+undersized+degraded, acting [1,0] 2024-09-20T20:00:32.759 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-09-20T20:00:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='client.54401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3315212255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2703054489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514] boot 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='client.54401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3315212255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='client.54401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3315212255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2703054489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2703054489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514] boot 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: osd.4 [v2:172.21.15.196:6800/3271126514,v1:172.21.15.196:6801/3271126514] boot 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:33.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:33.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:33.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:34.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='client.54419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:34.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: pgmap v187: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='client.54419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: pgmap v187: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='client.54419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: pgmap v187: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:34.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.565 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.565 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.565 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.569 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:34.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:34.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:34.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:34.571 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.245 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.249 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:35.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:35.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: pgmap v189: 161 pgs: 8 peering, 16 active+undersized, 15 active+undersized+degraded, 122 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:00:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: pgmap v189: 161 pgs: 8 peering, 16 active+undersized, 15 active+undersized+degraded, 122 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: pgmap v189: 161 pgs: 8 peering, 16 active+undersized, 15 active+undersized+degraded, 122 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:36.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:36.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:36.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 39/636 objects degraded (6.132%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 39/636 objects degraded (6.132%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 39/636 objects degraded (6.132%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:00:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: pgmap v190: 161 pgs: 8 peering, 4 active+undersized, 3 active+undersized+degraded, 146 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9/636 objects degraded (1.415%) 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: pgmap v190: 161 pgs: 8 peering, 4 active+undersized, 3 active+undersized+degraded, 146 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9/636 objects degraded (1.415%) 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: pgmap v190: 161 pgs: 8 peering, 4 active+undersized, 3 active+undersized+degraded, 146 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9/636 objects degraded (1.415%) 2024-09-20T20:00:38.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:38.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:38 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:00:38.690+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:00:39.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:00:40.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/636 objects degraded (1.415%), 3 pgs degraded) 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-20T20:00:40.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/636 objects degraded (1.415%), 3 pgs degraded) 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-20T20:00:40.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/636 objects degraded (1.415%), 3 pgs degraded) 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:40.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:41.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:41.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:41.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:41.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:41 smithi196 ceph-mon[76349]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[100622]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:41 smithi184 ceph-mon[102844]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:42] "GET /metrics HTTP/1.1" 200 38790 "" "Prometheus/2.43.0" 2024-09-20T20:00:43.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:43.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:43.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:43.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:44.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[100622]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:44.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:44.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:44.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[102844]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:44.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:44.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:44 smithi196 ceph-mon[76349]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:00:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:46 smithi184 ceph-mon[100622]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-20T20:00:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:46 smithi184 ceph-mon[102844]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-20T20:00:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:46 smithi196 ceph-mon[76349]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-20T20:00:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:48 smithi184 ceph-mon[100622]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:48 smithi184 ceph-mon[102844]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:48 smithi196 ceph-mon[76349]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:49.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:48.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:00:49.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:48.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:49 smithi184 ceph-mon[100622]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:49 smithi184 ceph-mon[102844]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:00:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:49 smithi196 ceph-mon[76349]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:52.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:00:52] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.43.0" 2024-09-20T20:00:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:52 smithi184 ceph-mon[100622]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:52 smithi184 ceph-mon[102844]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:52 smithi196 ceph-mon[76349]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:53.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:53.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:54.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:54 smithi196 ceph-mon[76349]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:54.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:54.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:54 smithi184 ceph-mon[100622]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:54.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:54.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:54 smithi184 ceph-mon[102844]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:54.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:00:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:55 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:55 smithi196 ceph-mon[76349]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[102844]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:00:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[100622]: Upgrade: osd.4 is safe to restart 2024-09-20T20:00:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:00:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:00:56.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:56 smithi196 ceph-mon[76349]: Upgrade: Updating osd.4 2024-09-20T20:00:56.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:56 smithi196 ceph-mon[76349]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:56.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:56 smithi196 ceph-mon[76349]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[100622]: Upgrade: Updating osd.4 2024-09-20T20:00:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[100622]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[100622]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[102844]: Upgrade: Updating osd.4 2024-09-20T20:00:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[102844]: Deploying daemon osd.4 on smithi196 2024-09-20T20:00:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:56 smithi184 ceph-mon[102844]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:00:57.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:00:58.270 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:58 smithi196 systemd[1]: Stopping Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:00:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:58 smithi184 ceph-mon[100622]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:58 smithi184 ceph-mon[102844]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:58 smithi196 ceph-mon[76349]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:00:58.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:58 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:58.304+0000 7fd4ad04e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:00:58.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:58 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:58.304+0000 7fd4ad04e640 -1 osd.4 141 *** Got signal Terminated *** 2024-09-20T20:00:58.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:00:58 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[91758]: 2024-09-20T20:00:58.304+0000 7fd4ad04e640 -1 osd.4 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:00:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:00:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:00:58.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:00:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:59 smithi184 ceph-mon[102844]: osd.4 marked itself down and dead 2024-09-20T20:00:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:59 smithi184 ceph-mon[100622]: osd.4 marked itself down and dead 2024-09-20T20:00:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:00:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:59 smithi196 ceph-mon[76349]: osd.4 marked itself down and dead 2024-09-20T20:00:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:00.273 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:00 smithi196 podman[96164]: 2024-09-20 20:01:00.07340824 +0000 UTC m=+1.913264782 container died 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-20T20:01:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[100622]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[100622]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:01:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[102844]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:00 smithi184 ceph-mon[102844]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:01:00.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:00 smithi196 ceph-mon[76349]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:00.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:00 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:00.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:00 smithi196 ceph-mon[76349]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:01:00.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:00 smithi196 podman[96164]: 2024-09-20 20:01:00.579459442 +0000 UTC m=+2.419315939 container cleanup 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T20:01:00.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:00 smithi196 bash[96164]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4 2024-09-20T20:01:00.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:00 smithi196 podman[96216]: 2024-09-20 20:01:00.772532597 +0000 UTC m=+1.119130006 container remove 54917c2217ff42e3bb7f0a0ac09a885bd28f9785a487f3e4200ac7232444ee5c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True) 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.469 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.098719394 +0000 UTC m=+0.016270507 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:01.469 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.223481156 +0000 UTC m=+0.141032255 container create 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T20:01:01.470 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.467820813 +0000 UTC m=+0.385371914 container init 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T20:01:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:01:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:01.731 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.471473236 +0000 UTC m=+0.389024337 container start 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T20:01:01.732 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.568030748 +0000 UTC m=+0.485581868 container attach 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:01.732 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:01 smithi196 podman[96330]: 2024-09-20 20:01:01.66943236 +0000 UTC m=+0.586983472 container died 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T20:01:02.286 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:02] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.43.0" 2024-09-20T20:01:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:02 smithi184 ceph-mon[100622]: pgmap v204: 161 pgs: 30 peering, 24 stale+active+clean, 107 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:02 smithi184 ceph-mon[102844]: pgmap v204: 161 pgs: 30 peering, 24 stale+active+clean, 107 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:02 smithi196 ceph-mon[76349]: pgmap v204: 161 pgs: 30 peering, 24 stale+active+clean, 107 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:01:02.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 podman[96354]: 2024-09-20 20:01:02.246945015 +0000 UTC m=+0.568238826 container remove 2ca229b517980a3242865f8ca4424ee9fe604bf8b5730c5470bec447a3431a2a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:01:02.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-20T20:01:02.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 systemd[1]: Stopped Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:02.641 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.4.service: Consumed 1.909s CPU time. 2024-09-20T20:01:02.641 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 systemd[1]: Starting Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:01:02.951 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 podman[96467]: 2024-09-20 20:01:02.586122495 +0000 UTC m=+0.016050275 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:02.951 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 podman[96467]: 2024-09-20 20:01:02.714917858 +0000 UTC m=+0.144845732 container create 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T20:01:03.049 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 podman[96467]: 2024-09-20 20:01:02.950560039 +0000 UTC m=+0.380487829 container init 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:02 smithi196 podman[96467]: 2024-09-20 20:01:02.954026133 +0000 UTC m=+0.383953909 container start 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 podman[96467]: 2024-09-20 20:01:03.033881658 +0000 UTC m=+0.463809450 container attach 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:03.339 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 17m - - 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 17m - - 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (11m) 2s ago 13m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (12m) 3s ago 12m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:01:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5m) 2s ago 11m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5m) 3s ago 16m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6m) 2s ago 18m 515M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5m) 2s ago 18m 64.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4m) 3s ago 17m 51.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4m) 2s ago 17m 60.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (12m) 2s ago 12m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (12m) 3s ago 12m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3m) 2s ago 16m 49.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2m) 2s ago 16m 47.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (107s) 2s ago 15m 46.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (77s) 2s ago 15m 53.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (38s) 3s ago 15m 44.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 54917c2217ff 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (14m) 3s ago 14m 62.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 aa406d64037d 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (14m) 3s ago 14m 59.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T20:01:03.588 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (13m) 3s ago 13m 63.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:01:03.589 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6m) 3s ago 13m 55.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:01:03.589 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (11m) 2s ago 11m 81.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:01:03.589 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (11m) 3s ago 11m 82.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:01:03.589 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (11m) 2s ago 11m 86.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:01:03.589 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (11m) 3s ago 11m 85.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:01:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:03.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:01:03.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:01:03.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:01:03.640 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:01:03.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:03 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:03.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:03 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:03.660 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:03.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:03.660 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:03.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:03.906 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3, 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:01:03.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:03.908 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:01:03.908 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T20:01:03.908 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 9 2024-09-20T20:01:03.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:01:03.908 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate[96482]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:01:04.140 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 bash[96467]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:01:04.141 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:03 smithi196 podman[96467]: 2024-09-20 20:01:03.872359669 +0000 UTC m=+1.302287447 container died 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T20:01:04.183 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:01:04.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout: osd.4 (root=default,host=smithi196) is down 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 14/636 objects degraded (2.201%), 5 pgs degraded 2024-09-20T20:01:04.502 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-09-20T20:01:04.503 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-09-20T20:01:04.503 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-09-20T20:01:04.503 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-09-20T20:01:04.503 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1d is active+undersized+degraded, acting [1,0] 2024-09-20T20:01:04.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:04 smithi196 ceph-mon[76349]: from='client.54431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:04 smithi196 ceph-mon[76349]: pgmap v205: 161 pgs: 5 active+undersized, 30 peering, 21 stale+active+clean, 5 active+undersized+degraded, 100 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:01:04.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:04 smithi196 ceph-mon[76349]: from='client.54437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/5912689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[100622]: from='client.54431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[100622]: pgmap v205: 161 pgs: 5 active+undersized, 30 peering, 21 stale+active+clean, 5 active+undersized+degraded, 100 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:01:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[100622]: from='client.54437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/5912689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[102844]: from='client.54431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[102844]: pgmap v205: 161 pgs: 5 active+undersized, 30 peering, 21 stale+active+clean, 5 active+undersized+degraded, 100 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14/636 objects degraded (2.201%) 2024-09-20T20:01:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[102844]: from='client.54437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/5912689' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:04.889 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:04 smithi196 podman[96618]: 2024-09-20 20:01:04.604802211 +0000 UTC m=+0.725734391 container remove 919efe23b4b60c81cd015c99dca261014bb8baa610c255c53e3d9fe08e47faf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4-activate, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T20:01:04.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:04 smithi196 podman[96678]: 2024-09-20 20:01:04.805458844 +0000 UTC m=+0.102727967 container create 268ec6ab15965caf827987efa43770b04ce309d4bc4113b0fa24e62aad3d0b07 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-20T20:01:04.890 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:04 smithi196 podman[96678]: 2024-09-20 20:01:04.718698303 +0000 UTC m=+0.015967439 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:05.317 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:05 smithi196 podman[96678]: 2024-09-20 20:01:05.024972063 +0000 UTC m=+0.322241206 container init 268ec6ab15965caf827987efa43770b04ce309d4bc4113b0fa24e62aad3d0b07 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:05.317 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:05 smithi196 podman[96678]: 2024-09-20 20:01:05.028265456 +0000 UTC m=+0.325534587 container start 268ec6ab15965caf827987efa43770b04ce309d4bc4113b0fa24e62aad3d0b07 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T20:01:05.317 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-osd[96698]: -- 172.21.15.196:0/3325885211 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55844da79a40 con 0x55844e848400 2024-09-20T20:01:05.317 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:05 smithi196 bash[96678]: 268ec6ab15965caf827987efa43770b04ce309d4bc4113b0fa24e62aad3d0b07 2024-09-20T20:01:05.317 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:05 smithi196 systemd[1]: Started Ceph osd.4 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:05.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='client.54455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3532599164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:05.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='client.54455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3532599164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='client.54455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:05.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3532599164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:05.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:05.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: pgmap v206: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: pgmap v206: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: pgmap v206: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:06.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:06 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[96694]: 2024-09-20T20:01:06.697+0000 7fd3c8aad640 -1 Falling back to public interface 2024-09-20T20:01:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:07.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: pgmap v207: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: pgmap v207: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: pgmap v207: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:08.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:09.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.442 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[96694]: 2024-09-20T20:01:10.290+0000 7fd3c8aad640 -1 osd.4 141 log_to_monitors true 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: pgmap v208: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.446 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:10.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:10 smithi196 ceph-mon[76349]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: pgmap v208: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: pgmap v208: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:10.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[102844]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:10.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:10 smithi184 ceph-mon[100622]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:01:11.477 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-4[96694]: 2024-09-20T20:01:11.277+0000 7fd3bc84f640 -1 osd.4 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: osdmap e144: 8 total, 7 up, 8 in 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 108/636 objects degraded (16.981%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: osdmap e144: 8 total, 7 up, 8 in 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:01:11.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: osdmap e144: 8 total, 7 up, 8 in 2024-09-20T20:01:11.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:11.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:11.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:11.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:11.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:12] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.43.0" 2024-09-20T20:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: pgmap v209: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579] boot 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: pgmap v209: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: pgmap v209: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s; 108/636 objects degraded (16.981%) 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579] boot 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: osd.4 [v2:172.21.15.196:6800/278512579,v1:172.21.15.196:6801/278512579] boot 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:01:12.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:01:12.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:12.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:12.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:12.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:13.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:13.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:13.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:13.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: pgmap v212: 161 pgs: 7 peering, 36 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 99/636 objects degraded (15.566%) 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: pgmap v212: 161 pgs: 7 peering, 36 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 99/636 objects degraded (15.566%) 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: pgmap v212: 161 pgs: 7 peering, 36 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 99/636 objects degraded (15.566%) 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:14.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: pgmap v214: 161 pgs: 11 peering, 12 active+undersized, 12 active+undersized+degraded, 126 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/636 objects degraded (4.717%) 2024-09-20T20:01:15.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: pgmap v214: 161 pgs: 11 peering, 12 active+undersized, 12 active+undersized+degraded, 126 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/636 objects degraded (4.717%) 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: pgmap v214: 161 pgs: 11 peering, 12 active+undersized, 12 active+undersized+degraded, 126 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/636 objects degraded (4.717%) 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:15.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:15.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:15.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:15.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:15.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 99/636 objects degraded (15.566%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 99/636 objects degraded (15.566%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 99/636 objects degraded (15.566%), 26 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:16.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.549 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: pgmap v215: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:17.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: pgmap v215: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: pgmap v215: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:17.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/636 objects degraded (4.717%), 12 pgs degraded) 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/636 objects degraded (4.717%), 12 pgs degraded) 2024-09-20T20:01:18.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/636 objects degraded (4.717%), 12 pgs degraded) 2024-09-20T20:01:18.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:18.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:18.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:18.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:18.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:18.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.648 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:19 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:19.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:19.811 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:19 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:20.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:20.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:20.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:21 smithi196 ceph-mon[76349]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:22] "GET /metrics HTTP/1.1" 200 38639 "" "Prometheus/2.43.0" 2024-09-20T20:01:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[100622]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:01:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:21 smithi184 ceph-mon[102844]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:01:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:23.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:23.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: Upgrade: Updating osd.5 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[100622]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: Upgrade: Updating osd.5 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:23.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:23.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:23 smithi184 ceph-mon[102844]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: Upgrade: Updating osd.5 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:23 smithi196 ceph-mon[76349]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:01:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:26 smithi184 ceph-mon[100622]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:26 smithi184 ceph-mon[102844]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:26.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:26 smithi196 ceph-mon[76349]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:01:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:27.640 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:27 smithi196 systemd[1]: Stopping Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:01:28.139 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[35352]: 2024-09-20T20:01:27.723+0000 7f9c13cde640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:01:28.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[35352]: 2024-09-20T20:01:27.723+0000 7f9c13cde640 -1 osd.5 146 *** Got signal Terminated *** 2024-09-20T20:01:28.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:27 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[35352]: 2024-09-20T20:01:27.723+0000 7f9c13cde640 -1 osd.5 146 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:01:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:28 smithi184 ceph-mon[102844]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:28 smithi184 ceph-mon[102844]: osd.5 marked itself down and dead 2024-09-20T20:01:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:28 smithi184 ceph-mon[100622]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:28 smithi184 ceph-mon[100622]: osd.5 marked itself down and dead 2024-09-20T20:01:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:28 smithi196 ceph-mon[76349]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:01:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:28 smithi196 ceph-mon[76349]: osd.5 marked itself down and dead 2024-09-20T20:01:29.001 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:28 smithi196 podman[101612]: 2024-09-20 20:01:28.643102884 +0000 UTC m=+1.058965389 container died aa406d64037dd37c9a91a40c207c8f00fcc7913400ec2b74fd5765c26668b655 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T20:01:29.283 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:29 smithi196 podman[101612]: 2024-09-20 20:01:29.090508365 +0000 UTC m=+1.506370871 container cleanup aa406d64037dd37c9a91a40c207c8f00fcc7913400ec2b74fd5765c26668b655 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T20:01:29.283 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:29 smithi196 bash[101612]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5 2024-09-20T20:01:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[102844]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:01:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[100622]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:29.640 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:29 smithi196 podman[101627]: 2024-09-20 20:01:29.38371192 +0000 UTC m=+0.740241603 container remove aa406d64037dd37c9a91a40c207c8f00fcc7913400ec2b74fd5765c26668b655 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-20T20:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:29 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:29 smithi196 ceph-mon[76349]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:30.038 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:29 smithi196 podman[101717]: 2024-09-20 20:01:29.618638719 +0000 UTC m=+0.015906751 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:30.038 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:29 smithi196 podman[101717]: 2024-09-20 20:01:29.751315217 +0000 UTC m=+0.148583240 container create a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:01:30.294 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 podman[101717]: 2024-09-20 20:01:30.037192935 +0000 UTC m=+0.434460969 container init a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T20:01:30.294 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 podman[101717]: 2024-09-20 20:01:30.040768412 +0000 UTC m=+0.438036442 container start a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:30.294 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 podman[101717]: 2024-09-20 20:01:30.120326192 +0000 UTC m=+0.517594218 container attach a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True) 2024-09-20T20:01:30.294 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 podman[101717]: 2024-09-20 20:01:30.223276791 +0000 UTC m=+0.620544821 container died a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:01:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[102844]: pgmap v222: 161 pgs: 10 peering, 17 stale+active+clean, 134 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:01:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[102844]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:01:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[102844]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:01:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[100622]: pgmap v222: 161 pgs: 10 peering, 17 stale+active+clean, 134 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:01:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[100622]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:01:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:30 smithi184 ceph-mon[100622]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:01:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:30 smithi196 ceph-mon[76349]: pgmap v222: 161 pgs: 10 peering, 17 stale+active+clean, 134 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:01:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:30 smithi196 ceph-mon[76349]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:01:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:30 smithi196 ceph-mon[76349]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:01:31.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 podman[101740]: 2024-09-20 20:01:30.75741728 +0000 UTC m=+0.524723128 container remove a0c5f7cdf9c7f763b1b60bc54bdb267ace67e8b1b1a7044522fe63108e1667d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T20:01:31.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T20:01:31.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 systemd[1]: Stopped Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:31.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.5.service: Consumed 9.456s CPU time. 2024-09-20T20:01:31.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:30 smithi196 systemd[1]: Starting Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:01:31.512 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:31 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 25/636 objects degraded (3.931%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:31.512 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 podman[101842]: 2024-09-20 20:01:31.102849769 +0000 UTC m=+0.016275111 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:31.512 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 podman[101842]: 2024-09-20 20:01:31.2340209 +0000 UTC m=+0.147446242 container create eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:01:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:31 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 25/636 objects degraded (3.931%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:31 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 25/636 objects degraded (3.931%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 podman[101842]: 2024-09-20 20:01:31.51160202 +0000 UTC m=+0.425027365 container init eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 podman[101842]: 2024-09-20 20:01:31.515243735 +0000 UTC m=+0.428669088 container start eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 podman[101842]: 2024-09-20 20:01:31.594931736 +0000 UTC m=+0.508357082 container attach eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 bash[101842]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:31.890 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:31 smithi196 bash[101842]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:32] "GET /metrics HTTP/1.1" 200 38779 "" "Prometheus/2.43.0" 2024-09-20T20:01:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-mon[76349]: pgmap v224: 161 pgs: 12 active+undersized, 10 peering, 11 stale+active+clean, 8 active+undersized+degraded, 120 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 25/636 objects degraded (3.931%) 2024-09-20T20:01:32.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:32.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:32.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:32.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:32 smithi184 ceph-mon[102844]: pgmap v224: 161 pgs: 12 active+undersized, 10 peering, 11 stale+active+clean, 8 active+undersized+degraded, 120 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 25/636 objects degraded (3.931%) 2024-09-20T20:01:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:32 smithi184 ceph-mon[100622]: pgmap v224: 161 pgs: 12 active+undersized, 10 peering, 11 stale+active+clean, 8 active+undersized+degraded, 120 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 25/636 objects degraded (3.931%) 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:01:32.800 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:32.801 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:32.801 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[101857]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:01:32.801 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 bash[101842]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:01:32.801 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:32 smithi196 podman[101842]: 2024-09-20 20:01:32.459011884 +0000 UTC m=+1.372437225 container died eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T20:01:33.541 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 podman[101998]: 2024-09-20 20:01:33.191109813 +0000 UTC m=+0.722297276 container remove eb87a6ff54f1a52e7fe202a77b6c1ec15ad560498f1058af16bc722d4013655d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:33.541 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 podman[102059]: 2024-09-20 20:01:33.300757695 +0000 UTC m=+0.015777700 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:33.541 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 podman[102059]: 2024-09-20 20:01:33.43379212 +0000 UTC m=+0.148812124 container create 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T20:01:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:33.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:33.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:34.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 podman[102059]: 2024-09-20 20:01:33.65297572 +0000 UTC m=+0.367995743 container init 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:01:34.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 podman[102059]: 2024-09-20 20:01:33.656237625 +0000 UTC m=+0.371257630 container start 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T20:01:34.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 ceph-osd[102079]: -- 172.21.15.196:0/2251470180 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bedef27a40 con 0x55bedef05c00 2024-09-20T20:01:34.141 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 bash[102059]: 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e 2024-09-20T20:01:34.141 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:33 smithi196 systemd[1]: Started Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: pgmap v225: 161 pgs: 29 active+undersized, 10 peering, 2 stale+active+clean, 14 active+undersized+degraded, 106 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 52/636 objects degraded (8.176%) 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: pgmap v225: 161 pgs: 29 active+undersized, 10 peering, 2 stale+active+clean, 14 active+undersized+degraded, 106 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 52/636 objects degraded (8.176%) 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: pgmap v225: 161 pgs: 29 active+undersized, 10 peering, 2 stale+active+clean, 14 active+undersized+degraded, 106 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 52/636 objects degraded (8.176%) 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:34.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:34.829 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:01:34.882 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:34 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:34.600+0000 7f050a8fb640 -1 Falling back to public interface 2024-09-20T20:01:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:01:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 18m - - 2024-09-20T20:01:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 18m - - 2024-09-20T20:01:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (12m) 14s ago 13m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:01:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (13m) 15s ago 13m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6m) 14s ago 11m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6m) 15s ago 17m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7m) 14s ago 19m 517M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5m) 14s ago 19m 67.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4m) 15s ago 17m 55.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5m) 14s ago 17m 63.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (13m) 14s ago 13m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (13m) 15s ago 13m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3m) 14s ago 17m 50.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2m) 14s ago 16m 49.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2m) 14s ago 16m 47.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (109s) 14s ago 15m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (30s) 15s ago 15m 43.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 starting - - - 4096M 2024-09-20T20:01:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (14m) 15s ago 14m 59.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (14m) 15s ago 14m 64.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6m) 15s ago 13m 56.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (12m) 14s ago 12m 82.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (12m) 15s ago 12m 83.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (11m) 14s ago 11m 86.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:01:35.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (11m) 15s ago 11m 85.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:01:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:35.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:35.809 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:01:35.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 5 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:01:35.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:01:35.811 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:01:35.811 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T20:01:35.811 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 10 2024-09-20T20:01:35.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:01:35.811 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:01:36.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:01:36.108 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout: osd.5 (root=default,host=smithi196) is down 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded 2024-09-20T20:01:36.444 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-09-20T20:01:36.445 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-09-20T20:01:36.446 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.1f is active+undersized+degraded, acting [6,1] 2024-09-20T20:01:36.446 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.c is active+undersized+degraded, acting [3,6] 2024-09-20T20:01:36.446 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-09-20T20:01:36.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='client.44517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: pgmap v226: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2264996315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='client.44517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: pgmap v226: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2264996315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='client.44511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='client.44517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: pgmap v226: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2264996315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:36.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1744026451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1744026451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1744026451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:37.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.357 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:37 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:37.927+0000 7f050a8fb640 -1 osd.5 146 log_to_monitors true 2024-09-20T20:01:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: pgmap v227: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: pgmap v227: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: pgmap v227: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:38.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:01:39.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: osdmap e149: 8 total, 7 up, 8 in 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: osdmap e149: 8 total, 7 up, 8 in 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.639 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:39.413+0000 7f04fe69d640 -1 osd.5 146 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: osdmap e149: 8 total, 7 up, 8 in 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:39.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: pgmap v229: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: pgmap v229: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: pgmap v229: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 71/636 objects degraded (11.164%) 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:01:40.813 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:01:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514] boot 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:01:41.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:01:41.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:41.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514] boot 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:41.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Upgrade: osd.5 is safe to restart 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: osd.5 [v2:172.21.15.196:6808/3689110514,v1:172.21.15.196:6809/3689110514] boot 2024-09-20T20:01:41.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:01:41.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:01:41.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:41.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:41.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:42] "GET /metrics HTTP/1.1" 200 38779 "" "Prometheus/2.43.0" 2024-09-20T20:01:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:42 smithi196 ceph-mon[76349]: pgmap v231: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:42 smithi196 ceph-mon[76349]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T20:01:42.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:42 smithi184 ceph-mon[100622]: pgmap v231: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:42.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:42 smithi184 ceph-mon[100622]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T20:01:42.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:42 smithi184 ceph-mon[102844]: pgmap v231: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:42.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:42 smithi184 ceph-mon[102844]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T20:01:43.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:43 smithi196 ceph-mon[76349]: pgmap v233: 161 pgs: 6 peering, 29 active+undersized, 16 active+undersized+degraded, 110 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/636 objects degraded (9.277%) 2024-09-20T20:01:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:43 smithi184 ceph-mon[100622]: pgmap v233: 161 pgs: 6 peering, 29 active+undersized, 16 active+undersized+degraded, 110 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/636 objects degraded (9.277%) 2024-09-20T20:01:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:43 smithi184 ceph-mon[102844]: pgmap v233: 161 pgs: 6 peering, 29 active+undersized, 16 active+undersized+degraded, 110 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/636 objects degraded (9.277%) 2024-09-20T20:01:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:43.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:43.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:44.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:44.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:45 smithi184 ceph-mon[102844]: pgmap v234: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 326 B/s rd, 0 op/s 2024-09-20T20:01:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:45 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:45 smithi184 ceph-mon[100622]: pgmap v234: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 326 B/s rd, 0 op/s 2024-09-20T20:01:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:45 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:46.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:45 smithi196 ceph-mon[76349]: pgmap v234: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 326 B/s rd, 0 op/s 2024-09-20T20:01:46.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:45 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded) 2024-09-20T20:01:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: Upgrade: Updating osd.5 2024-09-20T20:01:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[100622]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded) 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: Upgrade: Updating osd.5 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:46 smithi184 ceph-mon[102844]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/636 objects degraded (9.277%), 16 pgs degraded) 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: Upgrade: Updating osd.5 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:46 smithi196 ceph-mon[76349]: Deploying daemon osd.5 on smithi196 2024-09-20T20:01:47.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:47 smithi196 ceph-mon[76349]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:01:47.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:47 smithi184 ceph-mon[100622]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:01:47.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:47 smithi184 ceph-mon[102844]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:01:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:48.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:01:49.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:48.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:49.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:49 smithi196 systemd[1]: Stopping Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:01:49.798 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:49 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:49.524+0000 7f0507893640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:01:49.798 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:49 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:49.524+0000 7f0507893640 -1 osd.5 151 *** Got signal Terminated *** 2024-09-20T20:01:49.798 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:49 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[102075]: 2024-09-20T20:01:49.524+0000 7f0507893640 -1 osd.5 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:01:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:49 smithi184 ceph-mon[100622]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T20:01:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:49 smithi184 ceph-mon[100622]: osd.5 marked itself down and dead 2024-09-20T20:01:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:49 smithi184 ceph-mon[102844]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T20:01:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:49 smithi184 ceph-mon[102844]: osd.5 marked itself down and dead 2024-09-20T20:01:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:49 smithi196 ceph-mon[76349]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-09-20T20:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:49 smithi196 ceph-mon[76349]: osd.5 marked itself down and dead 2024-09-20T20:01:50.519 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:50 smithi196 podman[105059]: 2024-09-20 20:01:50.220613127 +0000 UTC m=+0.828037978 container died 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD) 2024-09-20T20:01:50.816 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:50 smithi196 podman[105059]: 2024-09-20 20:01:50.617632971 +0000 UTC m=+1.225057870 container cleanup 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:50.816 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:50 smithi196 bash[105059]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5 2024-09-20T20:01:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:50 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:51.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:50 smithi196 ceph-mon[76349]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:01:51.164 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:50 smithi196 podman[105075]: 2024-09-20 20:01:50.852155155 +0000 UTC m=+0.621937721 container remove 8a2bf92863127fef072c9821004af5b0681887fb3bd8648f2ed800e2a514fa1e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:01:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:50 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:50 smithi184 ceph-mon[100622]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:01:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:50 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:01:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:50 smithi184 ceph-mon[102844]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:01:51.490 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.089971866 +0000 UTC m=+0.016097201 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:51.490 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.228293894 +0000 UTC m=+0.154419260 container create 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:01:51.790 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.489095113 +0000 UTC m=+0.415220440 container init 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:01:51.790 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.492694045 +0000 UTC m=+0.418819369 container start 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T20:01:51.790 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.581022771 +0000 UTC m=+0.507148092 container attach 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:01:51.790 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:51 smithi196 podman[105164]: 2024-09-20 20:01:51.68602522 +0000 UTC m=+0.612150547 container died 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T20:01:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:51 smithi196 ceph-mon[76349]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:01:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:51 smithi196 ceph-mon[76349]: pgmap v239: 161 pgs: 5 peering, 22 stale+active+clean, 134 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-20T20:01:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:51 smithi184 ceph-mon[102844]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:01:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:51 smithi184 ceph-mon[102844]: pgmap v239: 161 pgs: 5 peering, 22 stale+active+clean, 134 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-20T20:01:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:51 smithi184 ceph-mon[100622]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:01:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:51 smithi184 ceph-mon[100622]: pgmap v239: 161 pgs: 5 peering, 22 stale+active+clean, 134 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-20T20:01:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:01:52] "GET /metrics HTTP/1.1" 200 38638 "" "Prometheus/2.43.0" 2024-09-20T20:01:52.536 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 podman[105190]: 2024-09-20 20:01:52.317716738 +0000 UTC m=+0.625213197 container remove 04646abf2b5bcfcb7cc6fb34b905e05443cc26149b1651692575434e7583b92c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:01:52.536 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T20:01:52.536 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 systemd[1]: Stopped Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:52.536 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.5.service: Consumed 1.986s CPU time. 2024-09-20T20:01:52.856 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 systemd[1]: Starting Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:01:52.856 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 podman[105299]: 2024-09-20 20:01:52.660935963 +0000 UTC m=+0.016023748 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:52.856 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 podman[105299]: 2024-09-20 20:01:52.76029617 +0000 UTC m=+0.115383947 container create 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T20:01:53.129 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 podman[105299]: 2024-09-20 20:01:52.996077995 +0000 UTC m=+0.351165779 container init 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:01:53.129 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:52 smithi196 podman[105299]: 2024-09-20 20:01:52.999413674 +0000 UTC m=+0.354501453 container start 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T20:01:53.129 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 podman[105299]: 2024-09-20 20:01:53.079348673 +0000 UTC m=+0.434436464 container attach 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T20:01:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:53.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:53.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:53.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:53.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:01:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:53 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:53 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:01:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:53.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:53.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:01:53.884 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:53.885 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:53.885 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:53.885 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:53.885 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate[105314]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 bash[105299]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:01:54.140 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:53 smithi196 podman[105299]: 2024-09-20 20:01:53.91107321 +0000 UTC m=+1.266160993 container died 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:01:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:54 smithi196 ceph-mon[76349]: pgmap v240: 161 pgs: 17 active+undersized, 5 peering, 13 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:01:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:54 smithi184 ceph-mon[100622]: pgmap v240: 161 pgs: 17 active+undersized, 5 peering, 13 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:01:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:54 smithi184 ceph-mon[102844]: pgmap v240: 161 pgs: 17 active+undersized, 5 peering, 13 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:54.964 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:54 smithi196 podman[105439]: 2024-09-20 20:01:54.658429399 +0000 UTC m=+0.737965314 container remove 72a9864b1d4822417b68b8f5efc9a654610888b478d91d9f5ae1d61ab7c586dc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T20:01:54.964 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:54 smithi196 podman[105497]: 2024-09-20 20:01:54.867708343 +0000 UTC m=+0.112231558 container create ba91460ce77dcb53675fdf678c1da8e199732c0b9ce4162d1d5e92e66efcf5ea (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:01:54.964 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:54 smithi196 podman[105497]: 2024-09-20 20:01:54.771701332 +0000 UTC m=+0.016224516 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:01:55.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:55 smithi196 podman[105497]: 2024-09-20 20:01:55.078359236 +0000 UTC m=+0.322882413 container init ba91460ce77dcb53675fdf678c1da8e199732c0b9ce4162d1d5e92e66efcf5ea (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:01:55.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:55 smithi196 podman[105497]: 2024-09-20 20:01:55.081659236 +0000 UTC m=+0.326182409 container start ba91460ce77dcb53675fdf678c1da8e199732c0b9ce4162d1d5e92e66efcf5ea (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T20:01:55.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:55 smithi196 ceph-osd[105515]: -- 172.21.15.196:0/466936806 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558a3c9cba40 con 0x558a3c9a9c00 2024-09-20T20:01:55.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:55 smithi196 bash[105497]: ba91460ce77dcb53675fdf678c1da8e199732c0b9ce4162d1d5e92e66efcf5ea 2024-09-20T20:01:55.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:55 smithi196 systemd[1]: Started Ceph osd.5 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:01:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: pgmap v241: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: pgmap v241: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: pgmap v241: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:56.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.083 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:01:56 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[105511]: 2024-09-20T20:01:56.955+0000 7fb1e3bc9640 -1 Falling back to public interface 2024-09-20T20:01:57.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:57.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.349 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:57.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: pgmap v242: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: pgmap v242: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: pgmap v242: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:58.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:58.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:01:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:01:58.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:01:59.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:01:59.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:01:59.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:01:59.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: pgmap v243: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: pgmap v243: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:00.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.640 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[105511]: 2024-09-20T20:02:00.277+0000 7fb1e3bc9640 -1 osd.5 151 log_to_monitors true 2024-09-20T20:02:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: pgmap v243: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:00.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.368 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-5[105511]: 2024-09-20T20:02:01.278+0000 7fb1db973640 -1 osd.5 151 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.373 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 71/636 objects degraded (11.164%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:01.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:01.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:01.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: pgmap v244: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: osdmap e154: 8 total, 7 up, 8 in 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:02] "GET /metrics HTTP/1.1" 200 38788 "" "Prometheus/2.43.0" 2024-09-20T20:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: pgmap v244: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: osdmap e154: 8 total, 7 up, 8 in 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: pgmap v244: 161 pgs: 40 active+undersized, 19 active+undersized+degraded, 102 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 71/636 objects degraded (11.164%) 2024-09-20T20:02:02.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: osdmap e154: 8 total, 7 up, 8 in 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:02.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470] boot 2024-09-20T20:02:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470] boot 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:02:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:03.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: osd.5 [v2:172.21.15.196:6808/2119017470,v1:172.21.15.196:6809/2119017470] boot 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:03.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:03.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:03.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:03.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:03.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: pgmap v247: 161 pgs: 8 peering, 33 active+undersized, 18 active+undersized+degraded, 102 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 61/636 objects degraded (9.591%) 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: pgmap v247: 161 pgs: 8 peering, 33 active+undersized, 18 active+undersized+degraded, 102 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 61/636 objects degraded (9.591%) 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: pgmap v247: 161 pgs: 8 peering, 33 active+undersized, 18 active+undersized+degraded, 102 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 61/636 objects degraded (9.591%) 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: pgmap v249: 161 pgs: 12 peering, 18 active+undersized, 10 active+undersized+degraded, 121 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 35/636 objects degraded (5.503%) 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 61/636 objects degraded (9.591%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: pgmap v249: 161 pgs: 12 peering, 18 active+undersized, 10 active+undersized+degraded, 121 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 35/636 objects degraded (5.503%) 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 61/636 objects degraded (9.591%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: pgmap v249: 161 pgs: 12 peering, 18 active+undersized, 10 active+undersized+degraded, 121 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 35/636 objects degraded (5.503%) 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 61/636 objects degraded (9.591%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:06.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:06.759 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 18m - - 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 18m - - 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (13m) 6s ago 14m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (13m) 18s ago 13m 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7m) 6s ago 12m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6m) 18s ago 17m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7m) 6s ago 19m 520M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6m) 6s ago 19m 69.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5m) 18s ago 18m 54.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:02:07.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5m) 6s ago 18m 64.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (13m) 6s ago 13m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (13m) 18s ago 13m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4m) 6s ago 17m 53.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3m) 6s ago 17m 51.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2m) 6s ago 16m 48.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2m) 6s ago 16m 56.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (62s) 18s ago 16m 44.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 starting - - - 4096M 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (15m) 18s ago 15m 61.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 57440ea827f8 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (14m) 18s ago 14m 64.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7m) 18s ago 14m 57.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (12m) 6s ago 12m 82.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (12m) 18s ago 12m 83.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (12m) 6s ago 12m 87.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:02:07.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (12m) 18s ago 12m 85.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:02:07.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:07.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:07.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:02:07.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 11 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:02:07.748 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:02:08.066 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:02:08.066 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:02:08.067 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:02:08.403 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:08.403 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:08.403 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='client.54521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='client.44562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: pgmap v250: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='client.54521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='client.44562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: pgmap v250: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/69883537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/69883537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:08.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/636 objects degraded (5.503%), 10 pgs degraded) 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/636 objects degraded (5.503%), 10 pgs degraded) 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.576 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='client.54521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='client.44562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.577 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: pgmap v250: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:08.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/69883537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/636 objects degraded (5.503%), 10 pgs degraded) 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:08.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:08.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:08.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/991340595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/991340595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/991340595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.560 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:09.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:09.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:09.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[100622]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[100622]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:02:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[102844]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:10 smithi184 ceph-mon[102844]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:02:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:10 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:10 smithi196 ceph-mon[76349]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:10 smithi196 ceph-mon[76349]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:02:12.289 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:12] "GET /metrics HTTP/1.1" 200 38788 "" "Prometheus/2.43.0" 2024-09-20T20:02:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:12 smithi184 ceph-mon[100622]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:02:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:12 smithi184 ceph-mon[102844]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:02:12.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:12 smithi196 ceph-mon[76349]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:02:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:13.543Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:13.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[100622]: Upgrade: Updating osd.6 2024-09-20T20:02:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[100622]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[102844]: Upgrade: Updating osd.6 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[102844]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:15 smithi196 ceph-mon[76349]: Upgrade: Updating osd.6 2024-09-20T20:02:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:15 smithi196 ceph-mon[76349]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:16 smithi196 ceph-mon[76349]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:16.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[102844]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[100622]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:17.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:17.818 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:17 smithi196 systemd[1]: Stopping Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:02:18.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:17 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[38795]: 2024-09-20T20:02:17.816+0000 7fe086b08640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:02:18.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:17 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[38795]: 2024-09-20T20:02:17.816+0000 7fe086b08640 -1 osd.6 156 *** Got signal Terminated *** 2024-09-20T20:02:18.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:17 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[38795]: 2024-09-20T20:02:17.816+0000 7fe086b08640 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:02:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:18 smithi196 ceph-mon[76349]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:02:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:18 smithi196 ceph-mon[76349]: osd.6 marked itself down and dead 2024-09-20T20:02:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-mon[100622]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:02:18.661 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-mon[100622]: osd.6 marked itself down and dead 2024-09-20T20:02:18.661 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-mon[102844]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:02:18.661 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-mon[102844]: osd.6 marked itself down and dead 2024-09-20T20:02:19.027 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:18 smithi196 podman[110018]: 2024-09-20 20:02:18.736718499 +0000 UTC m=+1.030693809 container died 57440ea827f89d9ef2df9578c29f84eaffe5da2f052e781fa974748f4522772b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:02:19.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:18.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:02:19.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:18.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:19.309 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110018]: 2024-09-20 20:02:19.116363993 +0000 UTC m=+1.410339295 container cleanup 57440ea827f89d9ef2df9578c29f84eaffe5da2f052e781fa974748f4522772b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T20:02:19.310 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 bash[110018]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6 2024-09-20T20:02:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:19 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:19 smithi196 ceph-mon[76349]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:02:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:19.640 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110036]: 2024-09-20 20:02:19.401243973 +0000 UTC m=+0.654697225 container remove 57440ea827f89d9ef2df9578c29f84eaffe5da2f052e781fa974748f4522772b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T20:02:19.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:19.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[100622]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:02:19.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:19.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:19.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[102844]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:02:19.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:19.989 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110123]: 2024-09-20 20:02:19.633169349 +0000 UTC m=+0.016031583 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:02:19.989 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110123]: 2024-09-20 20:02:19.752419679 +0000 UTC m=+0.135281894 container create 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T20:02:19.989 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110123]: 2024-09-20 20:02:19.988254801 +0000 UTC m=+0.371117030 container init 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T20:02:20.318 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:19 smithi196 podman[110123]: 2024-09-20 20:02:19.99168444 +0000 UTC m=+0.374546672 container start 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T20:02:20.318 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 podman[110123]: 2024-09-20 20:02:20.063088933 +0000 UTC m=+0.445951157 container attach 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T20:02:20.318 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 podman[110123]: 2024-09-20 20:02:20.177201384 +0000 UTC m=+0.560063619 container died 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T20:02:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:20 smithi196 ceph-mon[76349]: pgmap v257: 161 pgs: 11 peering, 12 stale+active+clean, 138 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-09-20T20:02:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:20 smithi196 ceph-mon[76349]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:02:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:20 smithi184 ceph-mon[102844]: pgmap v257: 161 pgs: 11 peering, 12 stale+active+clean, 138 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-09-20T20:02:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:20 smithi184 ceph-mon[102844]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:02:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:20 smithi184 ceph-mon[100622]: pgmap v257: 161 pgs: 11 peering, 12 stale+active+clean, 138 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-09-20T20:02:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:20 smithi184 ceph-mon[100622]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:02:21.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 podman[110146]: 2024-09-20 20:02:20.940029666 +0000 UTC m=+0.753694859 container remove 71355aa60bb23608df3c4aed00e34ad3f056c8f48fda53ec345cdf759e532479 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:02:21.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T20:02:21.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 systemd[1]: Stopped Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:02:21.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:20 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.6.service: Consumed 8.756s CPU time. 2024-09-20T20:02:21.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 systemd[1]: Starting Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:02:21.639 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 podman[110246]: 2024-09-20 20:02:21.278796452 +0000 UTC m=+0.016419577 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:02:21.640 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 podman[110246]: 2024-09-20 20:02:21.406143063 +0000 UTC m=+0.143766210 container create 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 podman[110246]: 2024-09-20 20:02:21.746377706 +0000 UTC m=+0.484000828 container init 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 podman[110246]: 2024-09-20 20:02:21.749849224 +0000 UTC m=+0.487472337 container start 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 podman[110246]: 2024-09-20 20:02:21.821341844 +0000 UTC m=+0.558964961 container attach 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 bash[110246]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:02:22.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:21 smithi196 bash[110246]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:02:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:22] "GET /metrics HTTP/1.1" 200 38779 "" "Prometheus/2.43.0" 2024-09-20T20:02:22.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-mon[76349]: pgmap v259: 161 pgs: 7 active+undersized, 11 peering, 9 stale+active+clean, 6 active+undersized+degraded, 128 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 19/636 objects degraded (2.987%) 2024-09-20T20:02:22.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 19/636 objects degraded (2.987%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:22.607 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:02:22.607 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:02:22.607 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:02:22.607 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:02:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[102844]: pgmap v259: 161 pgs: 7 active+undersized, 11 peering, 9 stale+active+clean, 6 active+undersized+degraded, 128 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 19/636 objects degraded (2.987%) 2024-09-20T20:02:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 19/636 objects degraded (2.987%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[100622]: pgmap v259: 161 pgs: 7 active+undersized, 11 peering, 9 stale+active+clean, 6 active+undersized+degraded, 128 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 19/636 objects degraded (2.987%) 2024-09-20T20:02:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:22 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 19/636 objects degraded (2.987%), 6 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:02:22.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:02:22.891 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:02:22.891 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[110261]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:02:22.891 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 bash[110246]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:02:22.891 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:22 smithi196 podman[110246]: 2024-09-20 20:02:22.685079645 +0000 UTC m=+1.422702758 container died 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:02:23.590 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 podman[110397]: 2024-09-20 20:02:23.250142843 +0000 UTC m=+0.553428569 container remove 02e0fbac77850ffa1eaf75ad9607cc7f5fc131649786987dbc07d152aec1fcf4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:02:23.590 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 podman[110466]: 2024-09-20 20:02:23.371822442 +0000 UTC m=+0.018050304 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:02:23.590 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 podman[110466]: 2024-09-20 20:02:23.484429291 +0000 UTC m=+0.130657143 container create dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:02:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:23.544Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:23.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:23.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 podman[110466]: 2024-09-20 20:02:23.695186916 +0000 UTC m=+0.341414778 container init dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T20:02:23.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 podman[110466]: 2024-09-20 20:02:23.698546168 +0000 UTC m=+0.344774023 container start dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:02:23.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 ceph-osd[110486]: -- 172.21.15.196:0/3329025116 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5568514b7a40 con 0x556852286400 2024-09-20T20:02:23.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 bash[110466]: dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 2024-09-20T20:02:23.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:23 smithi196 systemd[1]: Started Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:02:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: pgmap v260: 161 pgs: 25 active+undersized, 11 peering, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66/636 objects degraded (10.377%) 2024-09-20T20:02:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: pgmap v260: 161 pgs: 25 active+undersized, 11 peering, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66/636 objects degraded (10.377%) 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: pgmap v260: 161 pgs: 25 active+undersized, 11 peering, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 66/636 objects degraded (10.377%) 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:24.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:24.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.575 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.575 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.577 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.582 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.584 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:25.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:25.889 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:25 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:25.594+0000 7fc1269f3640 -1 Falling back to public interface 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: pgmap v261: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: pgmap v261: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: pgmap v261: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:26.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:27.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:27.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: pgmap v262: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: pgmap v262: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: pgmap v262: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:28.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:28.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:28.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:29.292 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:29.199+0000 7fc1269f3640 -1 osd.6 156 log_to_monitors true 2024-09-20T20:02:29.603 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.603 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:29.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.483 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:30.196+0000 7fc11e79d640 -1 osd.6 156 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: pgmap v263: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: osdmap e159: 8 total, 7 up, 8 in 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: pgmap v263: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: pgmap v263: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 78/636 objects degraded (12.264%) 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: osdmap e159: 8 total, 7 up, 8 in 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: osdmap e159: 8 total, 7 up, 8 in 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:30.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736] boot 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736] boot 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: osd.6 [v2:172.21.15.196:6816/174591736,v1:172.21.15.196:6817/174591736] boot 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:02:31.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:02:31.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:31.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:31.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:31.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:32] "GET /metrics HTTP/1.1" 200 38788 "" "Prometheus/2.43.0" 2024-09-20T20:02:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: pgmap v266: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: osdmap e161: 8 total, 8 up, 8 in 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: pgmap v266: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: pgmap v266: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:02:32.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: osdmap e161: 8 total, 8 up, 8 in 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: osdmap e161: 8 total, 8 up, 8 in 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:32.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:32.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:33.544Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:33.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:33.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:33.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:33.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:33.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: pgmap v268: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:02:34.337 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.338 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: pgmap v268: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: pgmap v268: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:34.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.588 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.590 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.591 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.592 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.593 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:35.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: pgmap v269: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:36.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:36.798 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:02:36.306+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T20:02:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: pgmap v269: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: pgmap v269: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:36.810 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:37 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T20:02:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:02:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:37.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:37 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T20:02:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:38 smithi196 ceph-mon[76349]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 435 B/s rd, 0 op/s 2024-09-20T20:02:38.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:38 smithi184 ceph-mon[100622]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 435 B/s rd, 0 op/s 2024-09-20T20:02:38.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:38 smithi184 ceph-mon[102844]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 435 B/s rd, 0 op/s 2024-09-20T20:02:38.696 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 19m - - 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 19m - - 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (13m) 17s ago 14m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (14m) 0s ago 14m 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7m) 17s ago 12m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7m) 0s ago 18m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (8m) 17s ago 20m 520M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6m) 17s ago 20m 71.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6m) 0s ago 18m 58.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6m) 17s ago 18m 65.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:02:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (14m) 17s ago 14m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (14m) 0s ago 14m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4m) 17s ago 18m 53.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4m) 17s ago 17m 52.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3m) 17s ago 17m 49.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2m) 17s ago 17m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (94s) 0s ago 16m 46.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (44s) 0s ago 16m 45.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (15s) 0s ago 15m 12.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 dd5f70360c21 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (15m) 0s ago 15m 64.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7m) 0s ago 15m 64.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (13m) 17s ago 13m 82.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (13m) 0s ago 13m 83.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (12m) 17s ago 12m 87.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:02:39.227 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (12m) 0s ago 12m 85.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:02:39.538 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:02:39.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:02:39.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T20:02:39.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 12 2024-09-20T20:02:39.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:02:39.540 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:02:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='client.54563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/288689832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='client.54563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-20T20:02:39.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/288689832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:39.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='client.54563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:39.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:39.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-20T20:02:39.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/288689832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "12/23 daemons upgraded", 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:02:39.814 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:02:39.815 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:02:40.169 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:40.169 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:40.169 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:40 smithi196 ceph-mon[76349]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4256734900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[100622]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4256734900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[102844]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4256734900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:02:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:41 smithi196 ceph-mon[76349]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:42] "GET /metrics HTTP/1.1" 200 38788 "" "Prometheus/2.43.0" 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[100622]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[102844]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-20T20:02:42.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:43.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:43.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:43.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[102844]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-20T20:02:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[100622]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-20T20:02:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:44 smithi196 ceph-mon[76349]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-09-20T20:02:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:46 smithi184 ceph-mon[100622]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:46 smithi184 ceph-mon[102844]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:46 smithi196 ceph-mon[76349]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:02:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:48 smithi184 ceph-mon[102844]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:48 smithi184 ceph-mon[100622]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:48 smithi196 ceph-mon[76349]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:48.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:48.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:02:48.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:48 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:48.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:50 smithi196 ceph-mon[76349]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:50 smithi184 ceph-mon[100622]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:50 smithi184 ceph-mon[102844]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:02:52] "GET /metrics HTTP/1.1" 200 38784 "" "Prometheus/2.43.0" 2024-09-20T20:02:52.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:52 smithi196 ceph-mon[76349]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:52.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:52.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:52 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:52.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:52 smithi196 ceph-mon[76349]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[100622]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[100622]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[102844]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:02:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:52 smithi184 ceph-mon[102844]: Upgrade: osd.6 is safe to restart 2024-09-20T20:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:02:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:02:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:53.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:53.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:54.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:54 smithi196 ceph-mon[76349]: Upgrade: Updating osd.6 2024-09-20T20:02:54.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:54 smithi196 ceph-mon[76349]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:54.484 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:54 smithi196 ceph-mon[76349]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[102844]: Upgrade: Updating osd.6 2024-09-20T20:02:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[102844]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[102844]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[100622]: Upgrade: Updating osd.6 2024-09-20T20:02:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[100622]: Deploying daemon osd.6 on smithi196 2024-09-20T20:02:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:54 smithi184 ceph-mon[100622]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:02:55.640 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:55 smithi196 systemd[1]: Stopping Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:02:56.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:55.750+0000 7fc12398b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:02:56.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:55.750+0000 7fc12398b640 -1 osd.6 161 *** Got signal Terminated *** 2024-09-20T20:02:56.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[110482]: 2024-09-20T20:02:55.751+0000 7fc12398b640 -1 osd.6 161 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:02:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:56 smithi196 ceph-mon[76349]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:56 smithi196 ceph-mon[76349]: osd.6 marked itself down and dead 2024-09-20T20:02:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[100622]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[100622]: osd.6 marked itself down and dead 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[102844]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:56 smithi184 ceph-mon[102844]: osd.6 marked itself down and dead 2024-09-20T20:02:57.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:57.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:57 smithi196 ceph-mon[76349]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T20:02:57.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:57 smithi196 ceph-mon[76349]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:02:57.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:57 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:57.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:57 smithi196 ceph-mon[76349]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:02:57.497 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:57 smithi196 podman[114935]: 2024-09-20 20:02:57.21663873 +0000 UTC m=+1.587343221 container died dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T20:02:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[100622]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T20:02:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[100622]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:02:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[100622]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:02:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:02:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[102844]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T20:02:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[102844]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:02:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:02:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:57 smithi184 ceph-mon[102844]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:02:57.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:57 smithi196 podman[114935]: 2024-09-20 20:02:57.676456958 +0000 UTC m=+2.047161438 container cleanup dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T20:02:57.891 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:57 smithi196 bash[114935]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6 2024-09-20T20:02:58.253 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:57 smithi196 podman[114966]: 2024-09-20 20:02:57.994728412 +0000 UTC m=+0.772643247 container remove dd5f70360c216be4a751802a8fc8123054741fdf329fbb145ed1cb199c047bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T20:02:58.639 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:58 smithi196 podman[114982]: 2024-09-20 20:02:58.423337267 +0000 UTC m=+0.722588424 container died f93227a674c4a4c087550939e265b0d54ceef27553dbd5ed3f0f97736b1b8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=romantic_morse, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:02:58.960 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:58 smithi196 ceph-mon[76349]: pgmap v282: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T20:02:58.960 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:58 smithi196 ceph-mon[76349]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:02:58.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:58.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:59.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:02:58.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:02:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[100622]: pgmap v282: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[100622]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[102844]: pgmap v282: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[102844]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:02:59.505 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.218691579 +0000 UTC m=+0.111251592 container create 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:02:59.506 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.123321082 +0000 UTC m=+0.015881067 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:02:59.506 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.437981119 +0000 UTC m=+0.330541090 container init 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:02:59.506 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.441875406 +0000 UTC m=+0.334435373 container start 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:02:59.782 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.504377431 +0000 UTC m=+0.396937414 container attach 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T20:02:59.783 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:02:59 smithi196 podman[115091]: 2024-09-20 20:02:59.631464255 +0000 UTC m=+0.524024229 container died 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:03:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:02:59 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 39/636 objects degraded (6.132%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:02:59 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 39/636 objects degraded (6.132%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:02:59 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 39/636 objects degraded (6.132%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:00.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 podman[115114]: 2024-09-20 20:03:00.266730918 +0000 UTC m=+0.629404554 container remove 3380b95eabad1e453563bc2b03d17690b1b315b49f99c6bb15c287fc8b967cc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T20:03:00.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T20:03:00.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 systemd[1]: Stopped Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:03:00.526 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.6.service: Consumed 1.842s CPU time. 2024-09-20T20:03:00.527 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 systemd[1]: Starting Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:03:00.783 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 podman[115212]: 2024-09-20 20:03:00.613416155 +0000 UTC m=+0.016055440 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:03:00.783 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 podman[115212]: 2024-09-20 20:03:00.72645314 +0000 UTC m=+0.129092400 container create 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T20:03:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:00 smithi184 ceph-mon[100622]: pgmap v284: 161 pgs: 15 active+undersized, 10 stale+active+clean, 11 active+undersized+degraded, 125 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:03:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:00 smithi184 ceph-mon[102844]: pgmap v284: 161 pgs: 15 active+undersized, 10 stale+active+clean, 11 active+undersized+degraded, 125 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:03:01.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:00 smithi196 ceph-mon[76349]: pgmap v284: 161 pgs: 15 active+undersized, 10 stale+active+clean, 11 active+undersized+degraded, 125 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/636 objects degraded (6.132%) 2024-09-20T20:03:01.054 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 podman[115212]: 2024-09-20 20:03:00.953649763 +0000 UTC m=+0.356289035 container init 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:03:01.055 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:00 smithi196 podman[115212]: 2024-09-20 20:03:00.957188762 +0000 UTC m=+0.359828035 container start 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T20:03:01.390 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 podman[115212]: 2024-09-20 20:03:01.053751724 +0000 UTC m=+0.456390987 container attach 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T20:03:01.390 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:01.390 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:01.390 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:01.390 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:03:01.849 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate[115228]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 bash[115212]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:03:02.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:01 smithi196 podman[115212]: 2024-09-20 20:03:01.876645399 +0000 UTC m=+1.279284660 container died 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T20:03:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:02] "GET /metrics HTTP/1.1" 200 38783 "" "Prometheus/2.43.0" 2024-09-20T20:03:02.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:02.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:02.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:02 smithi196 ceph-mon[76349]: pgmap v285: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 486 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:02.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:02 smithi196 ceph-mon[76349]: pgmap v286: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:02.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:02.875 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:02 smithi196 podman[115354]: 2024-09-20 20:03:02.658104793 +0000 UTC m=+0.774278423 container remove 0ae88c6c4b038114fcde8a5446131b473cfa812dfc1f30a1db98118ec69f9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T20:03:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[102844]: pgmap v285: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 486 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[102844]: pgmap v286: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[100622]: pgmap v285: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 486 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[100622]: pgmap v286: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:03.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:02 smithi196 podman[115413]: 2024-09-20 20:03:02.776216688 +0000 UTC m=+0.016368163 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:03:03.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:02 smithi196 podman[115413]: 2024-09-20 20:03:02.898812799 +0000 UTC m=+0.138964264 container create 82dcd5713f0fba59a156cdc248905337c6230393c981292c836c4a757ef2f49a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-20T20:03:03.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:03 smithi196 podman[115413]: 2024-09-20 20:03:03.119504916 +0000 UTC m=+0.359656394 container init 82dcd5713f0fba59a156cdc248905337c6230393c981292c836c4a757ef2f49a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T20:03:03.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:03 smithi196 podman[115413]: 2024-09-20 20:03:03.122581257 +0000 UTC m=+0.362732723 container start 82dcd5713f0fba59a156cdc248905337c6230393c981292c836c4a757ef2f49a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T20:03:03.440 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:03 smithi196 ceph-osd[115431]: -- 172.21.15.196:0/3484217183 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561c7414da40 con 0x561c74f1c000 2024-09-20T20:03:03.440 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:03 smithi196 bash[115413]: 82dcd5713f0fba59a156cdc248905337c6230393c981292c836c4a757ef2f49a 2024-09-20T20:03:03.440 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:03 smithi196 systemd[1]: Started Ceph osd.6 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:03:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:03.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:03.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: pgmap v287: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: pgmap v287: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: pgmap v287: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:04.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.126 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:04 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[115427]: 2024-09-20T20:03:04.791+0000 7f2c62718640 -1 Falling back to public interface 2024-09-20T20:03:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:05.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: pgmap v288: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: pgmap v288: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: pgmap v288: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.560 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:06.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.556 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:07.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: pgmap v289: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: pgmap v289: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.377 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[115427]: 2024-09-20T20:03:08.118+0000 7f2c62718640 -1 osd.6 161 log_to_monitors true 2024-09-20T20:03:08.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: pgmap v289: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:08.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: osdmap e164: 8 total, 7 up, 8 in 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: osdmap e164: 8 total, 7 up, 8 in 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.409 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:09.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: osdmap e164: 8 total, 7 up, 8 in 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.481 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:09.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:09.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:09.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:09.833 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 20 20:03:09 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-6[115427]: 2024-09-20T20:03:09.567+0000 7f2c564ba640 -1 osd.6 161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: pgmap v291: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='osd.6 ' entity='osd.6' 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: pgmap v291: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='osd.6 ' entity='osd.6' 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: pgmap v291: 161 pgs: 32 active+undersized, 21 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 78/636 objects degraded (12.264%) 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='osd.6 ' entity='osd.6' 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:10.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:10.541 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 19m - - 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 19m - - 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (14m) 9s ago 15m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (14m) 15s ago 14m 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (8m) 9s ago 13m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7m) 15s ago 19m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (8m) 9s ago 20m 521M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7m) 9s ago 20m 74.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6m) 15s ago 19m 60.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6m) 9s ago 19m 68.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:03:11.120 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (14m) 9s ago 14m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (14m) 15s ago 14m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5m) 9s ago 18m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4m) 9s ago 18m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3m) 9s ago 17m 50.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3m) 9s ago 17m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2m) 15s ago 17m 47.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (76s) 15s ago 16m 46.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 starting - - - 4096M 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (16m) 15s ago 16m 65.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 cd144fa99cb8 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (8m) 15s ago 15m 64.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (13m) 9s ago 13m 83.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (13m) 15s ago 13m 83.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (13m) 9s ago 13m 87.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:03:11.121 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (13m) 15s ago 13m 86.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:03:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049] boot 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:03:11.450 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:11.451 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:03:11.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T20:03:11.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 12 2024-09-20T20:03:11.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:03:11.452 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049] boot 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: osd.6 [v2:172.21.15.196:6816/3440554049,v1:172.21.15.196:6817/3440554049] boot 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:03:11.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:11.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:12] "GET /metrics HTTP/1.1" 200 38783 "" "Prometheus/2.43.0" 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 78/636 objects degraded (12.264%), 21 pgs degraded 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-09-20T20:03:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.3 is active+undersized+degraded, acting [7,2] 2024-09-20T20:03:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-09-20T20:03:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='client.54602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: pgmap v293: 161 pgs: 16 peering, 22 active+undersized, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/636 objects degraded (8.176%) 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2554494786' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:12.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='client.54602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: pgmap v293: 161 pgs: 16 peering, 22 active+undersized, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/636 objects degraded (8.176%) 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2554494786' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: osdmap e166: 8 total, 8 up, 8 in 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.555 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3882561002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 52/636 objects degraded (8.176%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.556 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: osdmap e166: 8 total, 8 up, 8 in 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.559 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3882561002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 52/636 objects degraded (8.176%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='client.54602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: pgmap v293: 161 pgs: 16 peering, 22 active+undersized, 15 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/636 objects degraded (8.176%) 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2554494786' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: osdmap e166: 8 total, 8 up, 8 in 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3882561002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 52/636 objects degraded (8.176%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:12.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:12.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.310 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:03:12 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=sqlstore.transactions t=2024-09-20T20:03:12.948968392Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T20:03:13.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='client.54626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:13.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:13.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='client.54626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='client.54626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.807 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:13.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: pgmap v295: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: pgmap v295: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: pgmap v295: 161 pgs: 16 peering, 20 active+undersized, 13 active+undersized+degraded, 112 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/636 objects degraded (6.604%) 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:14.809 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.622 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:15.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:15 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:03:15.906+0000 7fef66d8f640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-20T20:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: pgmap v296: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 497 B/s rd, 0 op/s 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:03:16.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:16 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: pgmap v296: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 497 B/s rd, 0 op/s 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: pgmap v296: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 497 B/s rd, 0 op/s 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:03:16.804 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:16.805 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:03:16.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:16 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/636 objects degraded (6.604%), 13 pgs degraded) 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: pgmap v297: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:17 smithi196 ceph-mon[76349]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-20T20:03:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: pgmap v297: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[100622]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: pgmap v297: 161 pgs: 16 peering, 145 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 468 B/s rd, 0 op/s 2024-09-20T20:03:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:17 smithi184 ceph-mon[102844]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-20T20:03:18.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:18.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:03:18.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:18.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s 2024-09-20T20:03:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.819 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 440 B/s rd, 0 op/s 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:18.820 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:19 smithi184 ceph-mon[100622]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:19 smithi184 ceph-mon[102844]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:19 smithi196 ceph-mon[76349]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.6 on smithi196 is in unknown state']) 2024-09-20T20:03:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:20 smithi184 ceph-mon[100622]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T20:03:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:20 smithi184 ceph-mon[102844]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T20:03:20.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:20 smithi196 ceph-mon[76349]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T20:03:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:22] "GET /metrics HTTP/1.1" 200 38635 "" "Prometheus/2.43.0" 2024-09-20T20:03:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[100622]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-09-20T20:03:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[102844]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:22 smithi196 ceph-mon[76349]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-09-20T20:03:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:23.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:23.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:24 smithi196 ceph-mon[76349]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-09-20T20:03:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:24 smithi184 ceph-mon[100622]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-09-20T20:03:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:24 smithi184 ceph-mon[102844]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2024-09-20T20:03:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:26 smithi196 ceph-mon[76349]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-09-20T20:03:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:26 smithi184 ceph-mon[100622]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-09-20T20:03:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:26 smithi184 ceph-mon[102844]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-09-20T20:03:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[100622]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T20:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:29.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:28.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:03:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[102844]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T20:03:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:28 smithi196 ceph-mon[76349]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T20:03:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:30 smithi196 ceph-mon[76349]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:30 smithi184 ceph-mon[100622]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:30 smithi184 ceph-mon[102844]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[102844]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[102844]: Upgrade: osd.7 is safe to restart 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[100622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:31 smithi184 ceph-mon[100622]: Upgrade: osd.7 is safe to restart 2024-09-20T20:03:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:03:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:31 smithi196 ceph-mon[76349]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:03:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:31 smithi196 ceph-mon[76349]: Upgrade: osd.7 is safe to restart 2024-09-20T20:03:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:32] "GET /metrics HTTP/1.1" 200 38780 "" "Prometheus/2.43.0" 2024-09-20T20:03:33.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:32 smithi196 ceph-mon[76349]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:33.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:33.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:03:33.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[100622]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[102844]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:03:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:33.862 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:33.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:33.862 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:33.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:34.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:33 smithi196 ceph-mon[76349]: Upgrade: Updating osd.7 2024-09-20T20:03:34.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:33 smithi196 ceph-mon[76349]: Deploying daemon osd.7 on smithi196 2024-09-20T20:03:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-mon[100622]: Upgrade: Updating osd.7 2024-09-20T20:03:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-mon[100622]: Deploying daemon osd.7 on smithi196 2024-09-20T20:03:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-mon[102844]: Upgrade: Updating osd.7 2024-09-20T20:03:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:33 smithi184 ceph-mon[102844]: Deploying daemon osd.7 on smithi196 2024-09-20T20:03:34.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:34 smithi196 ceph-mon[76349]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:34 smithi184 ceph-mon[100622]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:34 smithi184 ceph-mon[102844]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:03:35.559 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:35 smithi196 systemd[1]: Stopping Ceph osd.7 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:03:35.870 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:35 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[42236]: 2024-09-20T20:03:35.556+0000 7f0e010e0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:03:35.870 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:35 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[42236]: 2024-09-20T20:03:35.556+0000 7f0e010e0640 -1 osd.7 166 *** Got signal Terminated *** 2024-09-20T20:03:35.870 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:35 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[42236]: 2024-09-20T20:03:35.556+0000 7f0e010e0640 -1 osd.7 166 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:03:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:35 smithi184 ceph-mon[100622]: osd.7 marked itself down and dead 2024-09-20T20:03:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:35 smithi184 ceph-mon[102844]: osd.7 marked itself down and dead 2024-09-20T20:03:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:35 smithi196 ceph-mon[76349]: osd.7 marked itself down and dead 2024-09-20T20:03:36.639 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:36 smithi196 podman[119941]: 2024-09-20 20:03:36.374756715 +0000 UTC m=+0.909371078 container died cd144fa99cb89b207fcef2849aba656966c3d6667a60f54f14180234a1da86cc (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d) 2024-09-20T20:03:37.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:36 smithi196 ceph-mon[76349]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:37.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:37.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:36 smithi196 ceph-mon[76349]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:03:37.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:36 smithi196 ceph-mon[76349]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:03:37.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:36 smithi196 ceph-mon[76349]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[100622]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[100622]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[100622]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[100622]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:03:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[102844]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:03:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[102844]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:03:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[102844]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:03:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:36 smithi184 ceph-mon[102844]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:03:37.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[119941]: 2024-09-20 20:03:37.015345077 +0000 UTC m=+1.549959482 container cleanup cd144fa99cb89b207fcef2849aba656966c3d6667a60f54f14180234a1da86cc (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T20:03:37.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 bash[119941]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7 2024-09-20T20:03:37.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[119957]: 2024-09-20 20:03:37.208360944 +0000 UTC m=+0.833503728 container remove cd144fa99cb89b207fcef2849aba656966c3d6667a60f54f14180234a1da86cc (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:03:37.866 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[120047]: 2024-09-20 20:03:37.516864495 +0000 UTC m=+0.015759648 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:03:37.866 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[120047]: 2024-09-20 20:03:37.61750777 +0000 UTC m=+0.116402915 container create c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T20:03:37.866 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[120047]: 2024-09-20 20:03:37.861932709 +0000 UTC m=+0.360827843 container init c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-20T20:03:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:37 smithi196 ceph-mon[76349]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:03:38.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[120047]: 2024-09-20 20:03:37.865222998 +0000 UTC m=+0.364118140 container start c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-09-20T20:03:38.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:37 smithi196 podman[120047]: 2024-09-20 20:03:37.970510927 +0000 UTC m=+0.469406062 container attach c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T20:03:38.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 podman[120047]: 2024-09-20 20:03:38.072147812 +0000 UTC m=+0.571042963 container died c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T20:03:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:37 smithi184 ceph-mon[100622]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:03:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:37 smithi184 ceph-mon[102844]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:03:39.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:38 smithi196 ceph-mon[76349]: pgmap v310: 161 pgs: 20 peering, 13 stale+active+clean, 128 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:03:39.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:38 smithi196 ceph-mon[76349]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-20T20:03:39.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.089 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 podman[120082]: 2024-09-20 20:03:38.749300177 +0000 UTC m=+0.671157962 container remove c173346196b556ae43d203f34a7e8c5c7fdf7957716cf3f8f700ac32f0985811 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T20:03:39.090 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-20T20:03:39.090 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 systemd[1]: Stopped Ceph osd.7 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:03:39.090 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 systemd[1]: ceph-4b598dd2-7788-11ef-baf5-efdc52797490@osd.7.service: Consumed 10.994s CPU time. 2024-09-20T20:03:39.090 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:38 smithi196 systemd[1]: Starting Ceph osd.7 for 4b598dd2-7788-11ef-baf5-efdc52797490... 2024-09-20T20:03:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[100622]: pgmap v310: 161 pgs: 20 peering, 13 stale+active+clean, 128 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:03:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[100622]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-20T20:03:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[102844]: pgmap v310: 161 pgs: 20 peering, 13 stale+active+clean, 128 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[102844]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:40.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:39 smithi196 ceph-mon[76349]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:40.277 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 podman[120229]: 2024-09-20 20:03:40.03536578 +0000 UTC m=+0.016807410 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:03:40.277 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 podman[120229]: 2024-09-20 20:03:40.166956165 +0000 UTC m=+0.148397777 container create af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T20:03:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:39 smithi184 ceph-mon[100622]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:39 smithi184 ceph-mon[102844]: Health check failed: Degraded data redundancy: 27/636 objects degraded (4.245%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 podman[120229]: 2024-09-20 20:03:40.394216983 +0000 UTC m=+0.375658595 container init af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 podman[120229]: 2024-09-20 20:03:40.397552909 +0000 UTC m=+0.378994522 container start af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 podman[120229]: 2024-09-20 20:03:40.47768439 +0000 UTC m=+0.459126002 container attach af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 bash[120229]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:40.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:40.543 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:40 smithi196 bash[120229]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:03:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:40 smithi184 ceph-mon[100622]: pgmap v311: 161 pgs: 24 active+undersized, 20 peering, 3 stale+active+clean, 12 active+undersized+degraded, 102 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:03:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:40 smithi184 ceph-mon[102844]: pgmap v311: 161 pgs: 24 active+undersized, 20 peering, 3 stale+active+clean, 12 active+undersized+degraded, 102 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:03:41.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:40 smithi196 ceph-mon[76349]: pgmap v311: 161 pgs: 24 active+undersized, 20 peering, 3 stale+active+clean, 12 active+undersized+degraded, 102 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 27/636 objects degraded (4.245%) 2024-09-20T20:03:41.294 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:03:41.294 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:03:41.294 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:03:41.295 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:03:41.295 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:03:41.295 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate[120244]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 bash[120229]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:03:41.568 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:41 smithi196 podman[120229]: 2024-09-20 20:03:41.321080823 +0000 UTC m=+1.302522447 container died af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T20:03:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:42] "GET /metrics HTTP/1.1" 200 38780 "" "Prometheus/2.43.0" 2024-09-20T20:03:42.357 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 podman[120369]: 2024-09-20 20:03:42.04191946 +0000 UTC m=+0.714914640 container remove af04920415eeef46b7b35d40605dcffc4d33860dbab8de4f2273d25ac9521bf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7-activate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:03:42.357 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 podman[120441]: 2024-09-20 20:03:42.241488631 +0000 UTC m=+0.104716975 container create 3e53ce58d0b38433afa6bdb7663384ea08fcac3cfe99c76ebea984f637c4bea4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:03:42.357 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 podman[120441]: 2024-09-20 20:03:42.152803416 +0000 UTC m=+0.016031783 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:03:42.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 podman[120441]: 2024-09-20 20:03:42.485315907 +0000 UTC m=+0.348544260 container init 3e53ce58d0b38433afa6bdb7663384ea08fcac3cfe99c76ebea984f637c4bea4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:03:42.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 podman[120441]: 2024-09-20 20:03:42.488700878 +0000 UTC m=+0.351929223 container start 3e53ce58d0b38433afa6bdb7663384ea08fcac3cfe99c76ebea984f637c4bea4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:03:42.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-osd[120460]: -- 172.21.15.196:0/1631819172 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c83de9a40 con 0x556c83dc7c00 2024-09-20T20:03:42.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 bash[120441]: 3e53ce58d0b38433afa6bdb7663384ea08fcac3cfe99c76ebea984f637c4bea4 2024-09-20T20:03:42.640 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:42 smithi196 systemd[1]: Started Ceph osd.7 for 4b598dd2-7788-11ef-baf5-efdc52797490. 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 20m - - 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 20m - - 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (14m) 1s ago 15m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (15m) 4s ago 15m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (8m) 1s ago 13m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (8m) 4s ago 19m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:03:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (9m) 1s ago 21m 522M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7m) 1s ago 21m 78.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7m) 4s ago 20m 60.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7m) 1s ago 19m 71.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (15m) 1s ago 15m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (15m) 4s ago 15m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5m) 1s ago 19m 55.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5m) 1s ago 18m 54.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4m) 1s ago 18m 50.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3m) 1s ago 18m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2m) 4s ago 17m 49.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (108s) 4s ago 17m 46.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (40s) 4s ago 16m 50.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:03:42.992 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 starting - - - 4096M 2024-09-20T20:03:42.993 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (8m) 4s ago 16m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:03:42.993 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (14m) 1s ago 14m 83.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 3a4d352f2801 2024-09-20T20:03:42.993 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (14m) 4s ago 14m 84.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:03:42.993 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (13m) 1s ago 13m 88.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 9fab03e0422d 2024-09-20T20:03:42.993 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (13m) 4s ago 13m 86.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: pgmap v312: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 8 pgs peering) 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: pgmap v312: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 8 pgs peering) 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: pgmap v312: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 8 pgs peering) 2024-09-20T20:03:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:43.323 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:03:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:03:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:03:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4, 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 12 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:03:43.324 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:03:43.638 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mon" 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "12/23 daemons upgraded", 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:03:43.639 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:03:43.639 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:43.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:43.639 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:43.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:43.993 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 85/636 objects degraded (13.365%), 29 pgs degraded 2024-09-20T20:03:43.993 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: osd.7 (root=default,host=smithi196) is down 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 85/636 objects degraded (13.365%), 29 pgs degraded 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-09-20T20:03:43.994 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-09-20T20:03:43.995 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 5.1 is active+undersized+degraded, acting [4,3] 2024-09-20T20:03:43.996 INFO:teuthology.orchestra.run.smithi184.stdout: pg 6.3 is active+undersized+degraded, acting [6,2] 2024-09-20T20:03:44.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[120456]: 2024-09-20T20:03:43.894+0000 7f7914d8b640 -1 Falling back to public interface 2024-09-20T20:03:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='client.54638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='client.44679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='client.54653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/153556050' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='client.54638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='client.44679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='client.54653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/153556050' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='client.54638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='client.44679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='client.54653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/153556050' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:44.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:44.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:44.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:44.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: pgmap v313: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1498291117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: pgmap v313: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1498291117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:45.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: pgmap v313: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1498291117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:03:45.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:45.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 85/636 objects degraded (13.365%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 85/636 objects degraded (13.365%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 85/636 objects degraded (13.365%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:46.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:46.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:46.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: pgmap v314: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: pgmap v314: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: pgmap v314: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:47.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:47.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[120456]: 2024-09-20T20:03:47.484+0000 7f7914d8b640 -1 osd.7 166 log_to_monitors true 2024-09-20T20:03:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:48.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.140 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-osd-7[120456]: 2024-09-20T20:03:48.904+0000 7f7908b2d640 -1 osd.7 166 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:03:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: pgmap v315: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:03:49.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: osdmap e169: 8 total, 7 up, 8 in 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='osd.7 ' entity='osd.7' 2024-09-20T20:03:49.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: pgmap v315: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: osdmap e169: 8 total, 7 up, 8 in 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: pgmap v315: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='osd.7 ' entity='osd.7' 2024-09-20T20:03:49.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:49.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: osdmap e169: 8 total, 7 up, 8 in 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:49.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='osd.7 ' entity='osd.7' 2024-09-20T20:03:49.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265] boot 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:03:50.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:03:50.166 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265] boot 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:03:50.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:50.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: osd.7 [v2:172.21.15.196:6824/1384306265,v1:172.21.15.196:6825/1384306265] boot 2024-09-20T20:03:50.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:03:50.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:03:50.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: pgmap v317: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: pgmap v317: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: pgmap v317: 161 pgs: 40 active+undersized, 29 active+undersized+degraded, 92 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 85/636 objects degraded (13.365%) 2024-09-20T20:03:51.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:03:51.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.139 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.140 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:51.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.214 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:03:52] "GET /metrics HTTP/1.1" 200 38864 "" "Prometheus/2.43.0" 2024-09-20T20:03:52.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:52.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: pgmap v320: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/636 objects degraded (9.434%) 2024-09-20T20:03:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: Health check update: Degraded data redundancy: 60/636 objects degraded (9.434%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: pgmap v320: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/636 objects degraded (9.434%) 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: pgmap v320: 161 pgs: 10 peering, 29 active+undersized, 19 active+undersized+degraded, 103 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/636 objects degraded (9.434%) 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: Health check update: Degraded data redundancy: 60/636 objects degraded (9.434%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: Health check update: Degraded data redundancy: 60/636 objects degraded (9.434%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:53.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:53.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:53.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:53.985 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:53.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:53.985 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:03:53.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.314 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.316 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:54.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:54.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:54.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:54.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: pgmap v321: 161 pgs: 10 peering, 24 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: pgmap v321: 161 pgs: 10 peering, 24 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: pgmap v321: 161 pgs: 10 peering, 24 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/636 objects degraded (8.805%) 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.304 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:55.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:55.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:55.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:55.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:55.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:56.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:56.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:56.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/636 objects degraded (8.805%), 16 pgs degraded) 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:03:57.145 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:03:57.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:03:57.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.303 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/636 objects degraded (8.805%), 16 pgs degraded) 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/636 objects degraded (8.805%), 16 pgs degraded) 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:03:57.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:03:57.309 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:03:57.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:03:57.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:03:57.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:03:57.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: Upgrade: Setting container_image for all osd 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: osdmap e172: 8 total, 8 up, 8 in 2024-09-20T20:03:58.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: Upgrade: Setting container_image for all osd 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: osdmap e172: 8 total, 8 up, 8 in 2024-09-20T20:03:58.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: Upgrade: Setting container_image for all osd 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: osdmap e172: 8 total, 8 up, 8 in 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:03:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:03:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: Upgrade: Setting container_image for all mds 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: Upgrade: Setting container_image for all mds 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:03:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: Upgrade: Setting container_image for all mds 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi184.ctohee", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:00 smithi184 ceph-mon[100622]: Upgrade: Updating rgw.foo.smithi184.ctohee (1/4) 2024-09-20T20:04:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:00 smithi184 ceph-mon[100622]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T20:04:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:00 smithi184 ceph-mon[102844]: Upgrade: Updating rgw.foo.smithi184.ctohee (1/4) 2024-09-20T20:04:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:00 smithi184 ceph-mon[102844]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T20:04:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:00 smithi196 ceph-mon[76349]: Upgrade: Updating rgw.foo.smithi184.ctohee (1/4) 2024-09-20T20:04:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:00 smithi196 ceph-mon[76349]: Deploying daemon rgw.foo.smithi184.ctohee on smithi184 2024-09-20T20:04:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:01 smithi196 ceph-mon[76349]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:01.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:01 smithi184 ceph-mon[100622]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:01.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:01 smithi184 ceph-mon[102844]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:02.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:02] "GET /metrics HTTP/1.1" 200 38854 "" "Prometheus/2.43.0" 2024-09-20T20:04:03.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[100622]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:03.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[102844]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:03.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:02 smithi196 ceph-mon[76349]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:03.804 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:03.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:03.804 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:03.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:04.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[100622]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[102844]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:04.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:04 smithi196 ceph-mon[76349]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:04:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[100622]: Upgrade: Updating rgw.smpl.smithi184.fllvrf (2/4) 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[100622]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[102844]: Upgrade: Updating rgw.smpl.smithi184.fllvrf (2/4) 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:05.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:05 smithi184 ceph-mon[102844]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T20:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:05 smithi196 ceph-mon[76349]: Upgrade: Updating rgw.smpl.smithi184.fllvrf (2/4) 2024-09-20T20:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi184.fllvrf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:05 smithi196 ceph-mon[76349]: Deploying daemon rgw.smpl.smithi184.fllvrf on smithi184 2024-09-20T20:04:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:06 smithi196 ceph-mon[76349]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T20:04:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:06 smithi184 ceph-mon[102844]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T20:04:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:06 smithi184 ceph-mon[100622]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T20:04:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:08 smithi196 ceph-mon[76349]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:04:09.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:08 smithi184 ceph-mon[100622]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:04:09.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:08 smithi184 ceph-mon[102844]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:04:10.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.176 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.176 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.176 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:11.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:10 smithi184 ceph-mon[100622]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 86 B/s wr, 85 op/s 2024-09-20T20:04:11.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:10 smithi184 ceph-mon[102844]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 86 B/s wr, 85 op/s 2024-09-20T20:04:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:10 smithi196 ceph-mon[76349]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 86 B/s wr, 85 op/s 2024-09-20T20:04:12.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:12.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:12.220 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:12] "GET /metrics HTTP/1.1" 200 38854 "" "Prometheus/2.43.0" 2024-09-20T20:04:12.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:12.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[100622]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 85 B/s wr, 105 op/s 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[102844]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 85 B/s wr, 105 op/s 2024-09-20T20:04:13.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:13.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:13.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:13 smithi196 ceph-mon[76349]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 85 B/s wr, 105 op/s 2024-09-20T20:04:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.zyivrs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:13.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:13.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:14.292 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:04:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:14 smithi196 ceph-mon[76349]: Upgrade: Updating rgw.foo.smithi196.zyivrs (3/4) 2024-09-20T20:04:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:14 smithi196 ceph-mon[76349]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T20:04:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[100622]: Upgrade: Updating rgw.foo.smithi196.zyivrs (3/4) 2024-09-20T20:04:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[100622]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T20:04:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[102844]: Upgrade: Updating rgw.foo.smithi196.zyivrs (3/4) 2024-09-20T20:04:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[102844]: Deploying daemon rgw.foo.smithi196.zyivrs on smithi196 2024-09-20T20:04:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 20m - - 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 20m - - 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (15m) 5s ago 16m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (16m) 17s ago 16m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (9m) 5s ago 14m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (9m) 17s ago 20m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (9m) 5s ago 21m 524M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (8m) 5s ago 21m 78.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7m) 17s ago 20m 63.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:04:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7m) 5s ago 20m 70.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (15m) 5s ago 15m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (15m) 17s ago 15m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6m) 5s ago 19m 56.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5m) 5s ago 19m 55.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4m) 5s ago 19m 51.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4m) 5s ago 18m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3m) 17s ago 18m 49.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2m) 17s ago 17m 48.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (72s) 17s ago 17m 51.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (32s) 17s ago 17m 53.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (9m) 17s ago 16m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (11s) 5s ago 14m - - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (14m) 17s ago 14m 84.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 42f833d8d48b 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 starting - - - - 2024-09-20T20:04:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (14m) 17s ago 14m 86.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 4f0ba822c928 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:04:15.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 15 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:04:15.134 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:04:15.336 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:15 smithi196 ceph-mon[76349]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 170 B/s wr, 118 op/s 2024-09-20T20:04:15.336 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:15 smithi196 ceph-mon[76349]: from='client.44763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:04:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "14/23 daemons upgraded", 2024-09-20T20:04:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:04:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:04:15.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:04:15.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:15 smithi184 ceph-mon[100622]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 170 B/s wr, 118 op/s 2024-09-20T20:04:15.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:15 smithi184 ceph-mon[100622]: from='client.44763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:15.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:15 smithi184 ceph-mon[102844]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 170 B/s wr, 118 op/s 2024-09-20T20:04:15.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:15 smithi184 ceph-mon[102844]: from='client.44763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:15.715 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:04:15.715 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:04:15.715 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:04:16.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:16 smithi196 ceph-mon[76349]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:16 smithi196 ceph-mon[76349]: from='client.44775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1974669276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:16.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1012465430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[102844]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[102844]: from='client.44775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1974669276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1012465430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[100622]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[100622]: from='client.44775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1974669276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1012465430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:17.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:17 smithi196 ceph-mon[76349]: from='client.54749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:17.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:17 smithi196 ceph-mon[76349]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-20T20:04:17.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:17.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:17.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[100622]: from='client.54749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[100622]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[102844]: from='client.54749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[102844]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.vsmvti", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:19 smithi196 ceph-mon[76349]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-09-20T20:04:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:19 smithi196 ceph-mon[76349]: Upgrade: Updating rgw.smpl.smithi196.vsmvti (4/4) 2024-09-20T20:04:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:19 smithi196 ceph-mon[76349]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T20:04:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[100622]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-09-20T20:04:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[100622]: Upgrade: Updating rgw.smpl.smithi196.vsmvti (4/4) 2024-09-20T20:04:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[100622]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T20:04:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[102844]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 127 op/s 2024-09-20T20:04:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[102844]: Upgrade: Updating rgw.smpl.smithi196.vsmvti (4/4) 2024-09-20T20:04:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:19 smithi184 ceph-mon[102844]: Deploying daemon rgw.smpl.smithi196.vsmvti on smithi196 2024-09-20T20:04:21.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:21 smithi196 ceph-mon[76349]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 153 op/s 2024-09-20T20:04:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:21 smithi184 ceph-mon[100622]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 153 op/s 2024-09-20T20:04:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:21 smithi184 ceph-mon[102844]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 255 B/s wr, 153 op/s 2024-09-20T20:04:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:22] "GET /metrics HTTP/1.1" 200 38845 "" "Prometheus/2.43.0" 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:23.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:23.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:23.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:04:23.470+0000 7fef58cf3640 -1 no module 'dashboard' 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:04:23.470+0000 7fef58cf3640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:04:23.470+0000 7fef58cf3640 -1 cephadm.serve: 2024-09-20T20:04:23.800 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:04:23.470+0000 7fef58cf3640 -1 Traceback (most recent call last): 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/module.py", line 866, in serve 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: serve.serve() 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 94, in serve 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: self.mgr.remote('dashboard', 'set_rgw_credentials') 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: File "/usr/share/ceph/mgr/mgr_module.py", line 2376, in remote 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: return self._ceph_dispatch_remote(module_name, method_name, 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ImportError: Module not found 2024-09-20T20:04:23.801 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:04:24.388+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:24.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: Checking dashboard <-> RGW credentials 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[100622]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: Checking dashboard <-> RGW credentials 2024-09-20T20:04:24.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:04:24.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:24 smithi184 ceph-mon[102844]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T20:04:24.725 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: Checking dashboard <-> RGW credentials 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:04:24.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:24 smithi196 ceph-mon[76349]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T20:04:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:25 smithi196 ceph-mon[76349]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:04:25.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:25 smithi184 ceph-mon[100622]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:04:25.660 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:25 smithi184 ceph-mon[102844]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:04:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:26 smithi184 ceph-mon[100622]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 182 op/s 2024-09-20T20:04:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:26 smithi184 ceph-mon[102844]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 182 op/s 2024-09-20T20:04:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:26.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:26 smithi196 ceph-mon[76349]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 182 op/s 2024-09-20T20:04:26.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:28 smithi184 ceph-mon[100622]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 157 op/s 2024-09-20T20:04:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:28 smithi184 ceph-mon[102844]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 157 op/s 2024-09-20T20:04:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:28 smithi196 ceph-mon[76349]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 157 op/s 2024-09-20T20:04:29.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:29.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:29.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:29.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:29.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:32] "GET /metrics HTTP/1.1" 200 38894 "" "Prometheus/2.43.0" 2024-09-20T20:04:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:33 smithi184 ceph-mon[100622]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 85 B/s wr, 119 op/s 2024-09-20T20:04:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:33 smithi184 ceph-mon[102844]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 85 B/s wr, 119 op/s 2024-09-20T20:04:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:33 smithi196 ceph-mon[76349]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 85 B/s wr, 119 op/s 2024-09-20T20:04:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:33.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:33.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:35 smithi184 ceph-mon[100622]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:35 smithi184 ceph-mon[102844]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:35 smithi196 ceph-mon[76349]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:37 smithi196 ceph-mon[76349]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:37 smithi184 ceph-mon[100622]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:37 smithi184 ceph-mon[102844]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:04:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:39 smithi196 ceph-mon[76349]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 37 op/s 2024-09-20T20:04:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:39 smithi184 ceph-mon[100622]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 37 op/s 2024-09-20T20:04:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:39 smithi184 ceph-mon[102844]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 37 op/s 2024-09-20T20:04:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:41 smithi196 ceph-mon[76349]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 85 B/s wr, 14 op/s 2024-09-20T20:04:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:41 smithi184 ceph-mon[100622]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 85 B/s wr, 14 op/s 2024-09-20T20:04:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:41 smithi184 ceph-mon[102844]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 85 B/s wr, 14 op/s 2024-09-20T20:04:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:42] "GET /metrics HTTP/1.1" 200 38894 "" "Prometheus/2.43.0" 2024-09-20T20:04:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:43 smithi196 ceph-mon[76349]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:43 smithi184 ceph-mon[100622]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:43 smithi184 ceph-mon[102844]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:43.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:43.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:44.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:45.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:45 smithi184 ceph-mon[100622]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:45.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:45 smithi184 ceph-mon[102844]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:45 smithi196 ceph-mon[76349]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:46.016 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:04:46.568 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:04:46.568 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 21m - - 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 21m - - 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (15m) 16s ago 16m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (16m) 16s ago 16m 91.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (9m) 16s ago 14m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (9m) 16s ago 20m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (10m) 16s ago 22m 525M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (8m) 16s ago 22m 79.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (8m) 16s ago 21m 64.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (8m) 16s ago 20m 72.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (16m) 16s ago 16m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (16m) 16s ago 16m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7m) 16s ago 20m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:04:46.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6m) 16s ago 19m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5m) 16s ago 19m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5m) 16s ago 19m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3m) 16s ago 18m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2m) 16s ago 18m 51.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (103s) 16s ago 18m 54.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (64s) 16s ago 17m 58.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (9m) 16s ago 17m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (43s) 16s ago 15m 77.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (30s) 16s ago 15m 75.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (36s) 16s ago 14m 77.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:04:46.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (24s) 16s ago 14m 77.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:04:46.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:04:46.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:04:46.919 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:04:46.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:04:46.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:04:46.919 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:04:47.212 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:04:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:04:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:04:47.214 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:04:47.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:47.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:47.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='client.44841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1358123921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:47.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='client.44841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1358123921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='client.44835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='client.44841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1358123921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:04:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:47.529 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:04:47.530 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:04:47.530 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:04:47.530 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:04:47.530 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:04:48.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:48 smithi196 ceph-mon[76349]: from='client.54809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:48 smithi196 ceph-mon[76349]: from='client.54821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3835675584' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[100622]: from='client.54809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[100622]: from='client.54821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3835675584' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[102844]: from='client.54809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[102844]: from='client.54821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:04:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3835675584' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:49 smithi196 ceph-mon[76349]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[102844]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[100622]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:04:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:51 smithi196 ceph-mon[76349]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:51 smithi184 ceph-mon[102844]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:51 smithi184 ceph-mon[100622]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:04:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:04:53.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:53 smithi196 ceph-mon[76349]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:53 smithi184 ceph-mon[100622]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:53 smithi184 ceph-mon[102844]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:53.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:04:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:04:53.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:04:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:55 smithi196 ceph-mon[76349]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:55 smithi184 ceph-mon[100622]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:55 smithi184 ceph-mon[102844]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:57 smithi196 ceph-mon[76349]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:57 smithi184 ceph-mon[100622]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:57 smithi184 ceph-mon[102844]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:04:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:04:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:59 smithi196 ceph-mon[76349]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:59 smithi184 ceph-mon[100622]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:04:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:59 smithi184 ceph-mon[102844]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:04:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:01 smithi196 ceph-mon[76349]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:01 smithi184 ceph-mon[100622]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:01 smithi184 ceph-mon[102844]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:05:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:03 smithi184 ceph-mon[100622]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:03 smithi184 ceph-mon[102844]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:03.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:03 smithi196 ceph-mon[76349]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:03.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:03.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:05 smithi196 ceph-mon[76349]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:05 smithi184 ceph-mon[100622]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:05 smithi184 ceph-mon[102844]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[102844]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.163 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[100622]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:07.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:07.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:07 smithi196 ceph-mon[76349]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:09 smithi196 ceph-mon[76349]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[100622]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[102844]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:11 smithi196 ceph-mon[76349]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:11 smithi184 ceph-mon[102844]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:11 smithi184 ceph-mon[100622]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:05:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:13 smithi196 ceph-mon[76349]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:13 smithi184 ceph-mon[100622]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:13 smithi184 ceph-mon[102844]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:13.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:13.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:15 smithi196 ceph-mon[76349]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:15 smithi184 ceph-mon[100622]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:15 smithi184 ceph-mon[102844]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:17 smithi196 ceph-mon[76349]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:17 smithi184 ceph-mon[100622]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:17 smithi184 ceph-mon[102844]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:17.819 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:05:18.345 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:05:18.345 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 22m - - 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 21m - - 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (16m) 11s ago 17m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (17m) 9s ago 17m 92.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (10m) 11s ago 15m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (10m) 9s ago 21m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (10m) 11s ago 22m 525M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (9m) 11s ago 22m 80.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (8m) 9s ago 21m 65.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (9m) 11s ago 21m 73.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (16m) 11s ago 16m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (16m) 9s ago 16m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:05:18.346 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7m) 11s ago 20m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6m) 11s ago 20m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6m) 11s ago 20m 54.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5m) 11s ago 19m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4m) 9s ago 19m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3m) 9s ago 18m 51.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2m) 9s ago 18m 55.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (96s) 9s ago 18m 58.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (10m) 9s ago 17m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (75s) 11s ago 15m 78.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (62s) 9s ago 15m 76.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (67s) 11s ago 15m 78.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:05:18.347 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (55s) 9s ago 15m 77.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:05:18.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:05:18.655 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:05:18.927 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:05:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:05:18.928 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:05:18.928 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:05:19.235 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:05:19.236 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:05:19.236 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:05:19.236 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:05:19.236 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:05:19.236 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[100622]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:19.236 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[100622]: from='client.54833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.236 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[100622]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.236 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[100622]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/837983851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[102844]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[102844]: from='client.54833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[102844]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[102844]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/837983851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:19 smithi196 ceph-mon[76349]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:19 smithi196 ceph-mon[76349]: from='client.54833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:19 smithi196 ceph-mon[76349]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:19 smithi196 ceph-mon[76349]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/837983851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:20 smithi196 ceph-mon[76349]: from='client.44895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/611820140' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:20 smithi184 ceph-mon[102844]: from='client.44895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/611820140' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:20 smithi184 ceph-mon[100622]: from='client.44895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/611820140' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:21 smithi196 ceph-mon[76349]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:21 smithi184 ceph-mon[100622]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:21 smithi184 ceph-mon[102844]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:05:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:23 smithi196 ceph-mon[76349]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:23 smithi184 ceph-mon[100622]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:23 smithi184 ceph-mon[102844]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:23.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:23.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:25.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:25 smithi196 ceph-mon[76349]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:25 smithi184 ceph-mon[100622]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:25 smithi184 ceph-mon[102844]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:27.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:27 smithi184 ceph-mon[100622]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:27.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:27.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:27 smithi184 ceph-mon[102844]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:27.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:27 smithi196 ceph-mon[76349]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:28.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[102844]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[100622]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:29 smithi196 ceph-mon[76349]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:31 smithi184 ceph-mon[100622]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:31 smithi184 ceph-mon[102844]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:31 smithi196 ceph-mon[76349]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T20:05:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:33 smithi184 ceph-mon[100622]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:33 smithi184 ceph-mon[102844]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:33 smithi196 ceph-mon[76349]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:33.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:33.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:35 smithi184 ceph-mon[100622]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:35 smithi184 ceph-mon[102844]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:35 smithi196 ceph-mon[76349]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:37 smithi184 ceph-mon[100622]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:37 smithi184 ceph-mon[102844]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:37 smithi196 ceph-mon[76349]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:39 smithi184 ceph-mon[100622]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:39 smithi184 ceph-mon[102844]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:39 smithi196 ceph-mon[76349]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:41 smithi184 ceph-mon[100622]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:41 smithi184 ceph-mon[102844]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:41 smithi196 ceph-mon[76349]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T20:05:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:43 smithi184 ceph-mon[100622]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:43 smithi184 ceph-mon[102844]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:43 smithi196 ceph-mon[76349]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:43.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:43.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:05:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:43.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:44.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:45.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:45 smithi196 ceph-mon[76349]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:45 smithi184 ceph-mon[100622]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:45 smithi184 ceph-mon[102844]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:47 smithi184 ceph-mon[100622]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:47 smithi184 ceph-mon[102844]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:47 smithi196 ceph-mon[76349]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[100622]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[102844]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.525 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:05:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:49 smithi196 ceph-mon[76349]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:05:50.053 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:05:50.053 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 22m - - 2024-09-20T20:05:50.053 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 22m - - 2024-09-20T20:05:50.053 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (16m) 2s ago 17m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (17m) 1s ago 17m 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (10m) 2s ago 15m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (10m) 1s ago 21m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (11m) 2s ago 23m 526M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (9m) 2s ago 23m 81.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (9m) 1s ago 22m 67.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (9m) 2s ago 21m 74.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (17m) 2s ago 17m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (17m) 1s ago 17m 16.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (8m) 2s ago 21m 58.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7m) 2s ago 21m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6m) 2s ago 20m 54.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6m) 2s ago 20m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:05:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4m) 1s ago 19m 55.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3m) 1s ago 19m 52.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2m) 1s ago 19m 55.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2m) 1s ago 18m 58.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (10m) 1s ago 18m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (107s) 2s ago 16m 78.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (93s) 1s ago 16m 76.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (99s) 2s ago 15m 78.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:05:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (87s) 1s ago 15m 78.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:05:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:05:50.364 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:05:50.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:05:50.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:05:50.364 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:05:50.633 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:05:50.633 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:05:50.634 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:05:50.945 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:05:50.945 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:05:50.946 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:05:50.946 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:05:50.946 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: from='client.54875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: from='client.54881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/19431885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/336419325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: from='client.54875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: from='client.54881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/19431885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/336419325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: from='client.54875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: from='client.54881 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/19431885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:05:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/336419325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:05:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:05:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:52 smithi196 ceph-mon[76349]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:52 smithi184 ceph-mon[100622]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:52 smithi184 ceph-mon[102844]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:05:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:53 smithi196 ceph-mon[76349]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:53 smithi184 ceph-mon[100622]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:53 smithi184 ceph-mon[102844]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:53.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:53.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:05:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:05:53.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:05:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:55 smithi196 ceph-mon[76349]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:55.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:55 smithi184 ceph-mon[100622]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:55.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:55 smithi184 ceph-mon[102844]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:05:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:57 smithi196 ceph-mon[76349]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:57 smithi184 ceph-mon[100622]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:57 smithi184 ceph-mon[102844]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:05:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:59 smithi196 ceph-mon[76349]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:59 smithi184 ceph-mon[100622]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:05:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:59 smithi184 ceph-mon[102844]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:05:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:01 smithi196 ceph-mon[76349]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:01.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:01 smithi184 ceph-mon[100622]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:01.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:01 smithi184 ceph-mon[102844]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T20:06:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:03 smithi196 ceph-mon[76349]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:03 smithi184 ceph-mon[100622]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:03 smithi184 ceph-mon[102844]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:03.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:03.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:03.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:05 smithi196 ceph-mon[76349]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:05.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:05 smithi184 ceph-mon[100622]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:05.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:05 smithi184 ceph-mon[102844]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:07 smithi196 ceph-mon[76349]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:07.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:07 smithi184 ceph-mon[100622]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:07.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:07.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:07 smithi184 ceph-mon[102844]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:07.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[100622]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:08 smithi184 ceph-mon[102844]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:08 smithi196 ceph-mon[76349]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:11 smithi184 ceph-mon[100622]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:11 smithi184 ceph-mon[102844]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:11 smithi196 ceph-mon[76349]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T20:06:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:13 smithi184 ceph-mon[102844]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:13 smithi184 ceph-mon[100622]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:13 smithi196 ceph-mon[76349]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:13.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:13.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:13.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:15 smithi184 ceph-mon[100622]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:15 smithi184 ceph-mon[102844]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:15 smithi196 ceph-mon[76349]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:17 smithi184 ceph-mon[100622]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:17 smithi184 ceph-mon[102844]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:17 smithi196 ceph-mon[76349]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:19 smithi184 ceph-mon[100622]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:19 smithi184 ceph-mon[102844]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:19 smithi196 ceph-mon[76349]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:21.244 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:06:21.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:21 smithi196 ceph-mon[76349]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:21 smithi184 ceph-mon[100622]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:21 smithi184 ceph-mon[102844]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 23m - - 2024-09-20T20:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 22m - - 2024-09-20T20:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (17m) 14s ago 18m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (18m) 12s ago 18m 89.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (11m) 14s ago 16m 62.5M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (11m) 12s ago 22m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (12m) 14s ago 23m 527M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (10m) 14s ago 24m 82.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (9m) 12s ago 22m 67.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (10m) 14s ago 22m 75.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (17m) 14s ago 17m 15.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (17m) 12s ago 17m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (8m) 14s ago 21m 58.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7m) 14s ago 21m 60.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7m) 14s ago 21m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6m) 14s ago 20m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:06:21.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5m) 12s ago 20m 55.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4m) 12s ago 19m 52.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3m) 12s ago 19m 55.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2m) 12s ago 19m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (11m) 12s ago 18m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2m) 14s ago 16m 78.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2m) 12s ago 16m 76.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2m) 14s ago 16m 78.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:06:21.772 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (119s) 12s ago 16m 78.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:06:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:22] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:06:22.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:06:22.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:06:22.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:06:22.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:06:22.108 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:06:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:06:22.381 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:06:22.381 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:22 smithi184 ceph-mon[100622]: from='client.44943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:22.381 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1449038450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:22.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:22 smithi184 ceph-mon[102844]: from='client.44943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:22.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1449038450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:22.460 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:22 smithi196 ceph-mon[76349]: from='client.44943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:22.460 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1449038450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:22.688 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:06:22.688 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:06:22.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:06:22.688 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:06:22.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:06:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[100622]: from='client.54911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[100622]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[100622]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[100622]: from='client.54929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1053581848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[102844]: from='client.54911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[102844]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[102844]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[102844]: from='client.54929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1053581848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:23.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:23 smithi196 ceph-mon[76349]: from='client.54911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:23 smithi196 ceph-mon[76349]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:23.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:23 smithi196 ceph-mon[76349]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:23 smithi196 ceph-mon[76349]: from='client.54929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:23.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1053581848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:23.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:23.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:23.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:25.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:25 smithi196 ceph-mon[76349]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:25 smithi184 ceph-mon[100622]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:25 smithi184 ceph-mon[102844]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:27 smithi184 ceph-mon[100622]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:27 smithi184 ceph-mon[102844]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:27.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:27 smithi196 ceph-mon[76349]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:27.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:28 smithi196 ceph-mon[76349]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[100622]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[102844]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:29.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:29.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:31 smithi184 ceph-mon[100622]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:31 smithi184 ceph-mon[102844]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:31.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:31 smithi196 ceph-mon[76349]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T20:06:33.561 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:33 smithi184 ceph-mon[100622]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:33.561 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:33 smithi184 ceph-mon[102844]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:33 smithi196 ceph-mon[76349]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:33.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:33.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:33.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:35 smithi196 ceph-mon[76349]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:35.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:35 smithi184 ceph-mon[100622]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:35.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:35 smithi184 ceph-mon[102844]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:37 smithi196 ceph-mon[76349]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:37 smithi184 ceph-mon[100622]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:37 smithi184 ceph-mon[102844]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:39 smithi196 ceph-mon[76349]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:39 smithi184 ceph-mon[100622]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:39 smithi184 ceph-mon[102844]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:41 smithi196 ceph-mon[76349]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:41 smithi184 ceph-mon[100622]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:41 smithi184 ceph-mon[102844]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T20:06:43.563 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:43 smithi184 ceph-mon[100622]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:43.563 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:43 smithi184 ceph-mon[102844]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:43.563 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:43.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:43 smithi196 ceph-mon[76349]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:43.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:43.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:06:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:45 smithi196 ceph-mon[76349]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:45.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:45 smithi184 ceph-mon[100622]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:45 smithi184 ceph-mon[102844]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:47 smithi196 ceph-mon[76349]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:47.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:47 smithi184 ceph-mon[100622]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:47.743 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:47.743 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:47 smithi184 ceph-mon[102844]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:47.743 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[100622]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[102844]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:48 smithi196 ceph-mon[76349]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:48.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:48.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:06:51.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:51 smithi196 ceph-mon[76349]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:51 smithi184 ceph-mon[100622]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:51 smithi184 ceph-mon[102844]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:06:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:06:52.988 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:06:53.516 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 23m - - 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 23m - - 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (17m) 5s ago 18m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (18m) 4s ago 18m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (11m) 5s ago 16m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (11m) 4s ago 22m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (12m) 5s ago 24m 527M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (10m) 5s ago 24m 83.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (10m) 4s ago 23m 66.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (10m) 5s ago 23m 75.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (18m) 5s ago 18m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (18m) 4s ago 18m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:06:53.517 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (9m) 5s ago 22m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (8m) 5s ago 22m 60.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7m) 5s ago 21m 54.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7m) 5s ago 21m 65.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5m) 4s ago 20m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4m) 4s ago 20m 52.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3m) 4s ago 20m 56.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3m) 4s ago 19m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (12m) 4s ago 19m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2m) 5s ago 17m 79.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2m) 4s ago 17m 77.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2m) 5s ago 17m 79.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:06:53.518 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2m) 4s ago 16m 79.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:06:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:53 smithi184 ceph-mon[100622]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:53 smithi184 ceph-mon[102844]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:53.561Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:53.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:06:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:06:53.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:06:53.827 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:06:53.828 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:06:53.829 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:06:53.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:06:53.829 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:06:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:53 smithi196 ceph-mon[76349]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:06:54.098 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:06:54.099 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:06:54.099 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:06:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:06:54.407 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:06:54.407 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:06:54.407 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:06:54.407 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:06:54.407 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:06:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[102844]: from='client.54941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[102844]: from='client.44985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4119812391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3020033638' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[100622]: from='client.54941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[100622]: from='client.44985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4119812391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3020033638' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:54 smithi196 ceph-mon[76349]: from='client.54941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:54 smithi196 ceph-mon[76349]: from='client.44985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4119812391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:06:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3020033638' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:06:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[100622]: from='client.44991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[100622]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[100622]: from='client.45003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[102844]: from='client.44991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[102844]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:55 smithi184 ceph-mon[102844]: from='client.45003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:55 smithi196 ceph-mon[76349]: from='client.44991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:55 smithi196 ceph-mon[76349]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:06:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:55 smithi196 ceph-mon[76349]: from='client.45003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:06:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:56 smithi184 ceph-mon[100622]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:56 smithi184 ceph-mon[102844]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:56 smithi196 ceph-mon[76349]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:06:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:58 smithi184 ceph-mon[100622]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:58 smithi184 ceph-mon[102844]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:06:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:58 smithi196 ceph-mon[76349]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:00 smithi184 ceph-mon[100622]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:00 smithi184 ceph-mon[102844]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:00 smithi196 ceph-mon[76349]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:07:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:02 smithi184 ceph-mon[100622]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:02.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:02 smithi184 ceph-mon[102844]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:02 smithi196 ceph-mon[76349]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:03.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:03.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:03.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:04 smithi184 ceph-mon[100622]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:04 smithi184 ceph-mon[102844]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:04 smithi196 ceph-mon[76349]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:06.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:06 smithi196 ceph-mon[76349]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:06.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:06 smithi184 ceph-mon[100622]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:06 smithi184 ceph-mon[102844]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:09.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[100622]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:09.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[102844]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:08 smithi196 ceph-mon[76349]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:09.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:09.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[100622]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:10 smithi184 ceph-mon[102844]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:10 smithi196 ceph-mon[76349]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:07:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:12 smithi184 ceph-mon[102844]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:12 smithi184 ceph-mon[100622]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:12 smithi196 ceph-mon[76349]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:14.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:13.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:13.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:13.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:14 smithi184 ceph-mon[102844]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:14 smithi184 ceph-mon[100622]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:14 smithi196 ceph-mon[76349]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:16 smithi184 ceph-mon[100622]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:16 smithi184 ceph-mon[102844]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:16 smithi196 ceph-mon[76349]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:19.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:18 smithi184 ceph-mon[100622]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:19.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:18 smithi184 ceph-mon[102844]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:18 smithi196 ceph-mon[76349]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:20 smithi184 ceph-mon[100622]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:20 smithi184 ceph-mon[102844]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:20 smithi196 ceph-mon[76349]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:22] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:07:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:22 smithi184 ceph-mon[100622]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:22 smithi184 ceph-mon[102844]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:22 smithi196 ceph-mon[76349]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:23.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:23.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:23.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:24.702 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:07:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:24 smithi184 ceph-mon[100622]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:24 smithi184 ceph-mon[102844]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:24 smithi196 ceph-mon[76349]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:25.229 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:07:25.229 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 24m - - 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 23m - - 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (18m) 17s ago 19m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (19m) 15s ago 19m 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (12m) 17s ago 17m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (12m) 15s ago 23m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (13m) 17s ago 24m 527M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (11m) 17s ago 25m 83.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (10m) 15s ago 23m 67.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (11m) 17s ago 23m 75.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (19m) 17s ago 18m 15.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (18m) 15s ago 18m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:07:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (9m) 17s ago 22m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (8m) 17s ago 22m 60.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (8m) 17s ago 22m 55.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7m) 17s ago 21m 65.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6m) 15s ago 21m 55.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5m) 15s ago 21m 52.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4m) 15s ago 20m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3m) 15s ago 20m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (12m) 15s ago 19m 65.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3m) 17s ago 17m 79.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3m) 15s ago 18m 77.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3m) 17s ago 17m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:07:25.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3m) 15s ago 17m 79.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:07:25.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:07:25.541 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:07:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[100622]: from='client.45015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[100622]: from='client.45021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[100622]: from='client.45027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1845589097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[102844]: from='client.45015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[102844]: from='client.45021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[102844]: from='client.45027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1845589097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:25.814 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:07:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:07:25.815 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:07:26.124 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:07:26.124 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:07:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:07:26.124 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:07:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:07:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:25 smithi196 ceph-mon[76349]: from='client.45015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:25 smithi196 ceph-mon[76349]: from='client.45021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:25 smithi196 ceph-mon[76349]: from='client.45027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1845589097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[100622]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[100622]: from='client.55001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2764103509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[102844]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:27.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:27.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[102844]: from='client.55001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:27.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2764103509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:26 smithi196 ceph-mon[76349]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:26 smithi196 ceph-mon[76349]: from='client.55001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2764103509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:28 smithi196 ceph-mon[76349]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[100622]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[102844]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[100622]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[102844]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:30 smithi196 ceph-mon[76349]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:07:33.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:32 smithi184 ceph-mon[100622]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:33.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:32 smithi184 ceph-mon[102844]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:32 smithi196 ceph-mon[76349]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:33.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:33.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:33.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:34 smithi184 ceph-mon[100622]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:34 smithi184 ceph-mon[102844]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:34 smithi196 ceph-mon[76349]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:36 smithi184 ceph-mon[100622]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:36 smithi184 ceph-mon[102844]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:36 smithi196 ceph-mon[76349]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:38 smithi184 ceph-mon[100622]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:38 smithi184 ceph-mon[102844]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:38 smithi196 ceph-mon[76349]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:40 smithi184 ceph-mon[100622]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:40 smithi184 ceph-mon[102844]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:40 smithi196 ceph-mon[76349]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:07:42.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:42 smithi184 ceph-mon[100622]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:42.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:42 smithi184 ceph-mon[102844]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:42 smithi196 ceph-mon[76349]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:43.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:43.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:43.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:44 smithi184 ceph-mon[100622]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:44 smithi184 ceph-mon[102844]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:44 smithi196 ceph-mon[76349]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:46 smithi184 ceph-mon[102844]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:46 smithi184 ceph-mon[100622]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:46 smithi196 ceph-mon[76349]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:49.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:49 smithi196 ceph-mon[76349]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:49.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[102844]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[100622]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:49.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:50 smithi196 ceph-mon[76349]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[100622]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[102844]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:07:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:07:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T20:07:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:52 smithi184 ceph-mon[100622]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:52 smithi184 ceph-mon[102844]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:52 smithi196 ceph-mon[76349]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:53.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:53.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:07:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:07:53.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:07:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:54 smithi184 ceph-mon[100622]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:54 smithi184 ceph-mon[102844]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:54 smithi196 ceph-mon[76349]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:07:56.418 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 24m - - 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 24m - - 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (18m) 8s ago 19m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (19m) 7s ago 19m 90.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (12m) 8s ago 17m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:07:56.948 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (12m) 7s ago 23m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (13m) 8s ago 25m 528M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (11m) 8s ago 25m 84.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (11m) 7s ago 24m 68.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (11m) 8s ago 24m 76.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (19m) 8s ago 19m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (19m) 7s ago 19m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (10m) 8s ago 23m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (9m) 8s ago 23m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (8m) 8s ago 22m 55.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (8m) 8s ago 22m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6m) 7s ago 21m 56.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6m) 7s ago 21m 53.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4m) 7s ago 21m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4m) 7s ago 20m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:07:56.949 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (13m) 7s ago 20m 65.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:07:56.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3m) 8s ago 18m 79.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:07:56.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3m) 7s ago 18m 77.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:07:56.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3m) 8s ago 18m 80.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:07:56.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3m) 7s ago 18m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:07:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[100622]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[100622]: from='client.55013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[102844]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:56 smithi184 ceph-mon[102844]: from='client.55013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:56 smithi196 ceph-mon[76349]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:56 smithi196 ceph-mon[76349]: from='client.55013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:07:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:07:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:07:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:07:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:07:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:07:57.260 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:07:57.529 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:07:57.530 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:07:57.530 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:07:57.530 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:07:57.530 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:07:57.530 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:07:57.845 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:07:57.845 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:07:57.845 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:07:57.846 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:07:57.846 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:07:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:57 smithi196 ceph-mon[76349]: from='client.55019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:57 smithi196 ceph-mon[76349]: from='client.55025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1011099125' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[100622]: from='client.55019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[100622]: from='client.55025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1011099125' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[102844]: from='client.55019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[102844]: from='client.55025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1011099125' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:07:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:58 smithi196 ceph-mon[76349]: from='client.45075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:58 smithi196 ceph-mon[76349]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3557442452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:07:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[100622]: from='client.45075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[100622]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3557442452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[102844]: from='client.45075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[102844]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3557442452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:07:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:00 smithi196 ceph-mon[76349]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:00 smithi184 ceph-mon[100622]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:00 smithi184 ceph-mon[102844]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:02] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:08:03.084 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:02 smithi184 ceph-mon[100622]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:03.085 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:02 smithi184 ceph-mon[102844]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:02 smithi196 ceph-mon[76349]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:03.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:03.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:03.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:04 smithi196 ceph-mon[76349]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:04 smithi184 ceph-mon[100622]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:04 smithi184 ceph-mon[102844]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:06 smithi196 ceph-mon[76349]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:06 smithi184 ceph-mon[100622]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:06 smithi184 ceph-mon[102844]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:09 smithi196 ceph-mon[76349]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[100622]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[102844]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:09.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[100622]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[102844]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.230 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:10 smithi196 ceph-mon[76349]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:12.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:12] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:08:13.151 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:12 smithi184 ceph-mon[100622]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:13.151 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:12 smithi184 ceph-mon[102844]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:12 smithi196 ceph-mon[76349]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:13.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:13.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:13.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:13.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:08:13.813131846Z level=info msg="Completed cleanup jobs" duration=13.036736ms 2024-09-20T20:08:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:14 smithi184 ceph-mon[100622]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:14 smithi184 ceph-mon[102844]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:14 smithi196 ceph-mon[76349]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:16 smithi184 ceph-mon[100622]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:16 smithi184 ceph-mon[102844]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:16 smithi196 ceph-mon[76349]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:18 smithi184 ceph-mon[100622]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:18 smithi184 ceph-mon[102844]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:18 smithi196 ceph-mon[76349]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:20 smithi184 ceph-mon[100622]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:20 smithi184 ceph-mon[102844]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:20 smithi196 ceph-mon[76349]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:22] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:08:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:22 smithi184 ceph-mon[100622]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:22 smithi184 ceph-mon[102844]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:23.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:22 smithi196 ceph-mon[76349]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:23.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:23.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:23.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:23.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:24 smithi184 ceph-mon[100622]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:24 smithi184 ceph-mon[102844]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:24 smithi196 ceph-mon[76349]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:26 smithi184 ceph-mon[100622]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:26 smithi184 ceph-mon[102844]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:26 smithi196 ceph-mon[76349]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:28.143 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 25m - - 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 24m - - 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (19m) 0s ago 20m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (20m) 18s ago 20m 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (13m) 0s ago 18m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (13m) 18s ago 24m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:08:28.697 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (14m) 0s ago 26m 528M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (12m) 0s ago 26m 86.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (11m) 18s ago 24m 68.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (12m) 0s ago 24m 78.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (20m) 0s ago 20m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (19m) 18s ago 19m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (10m) 0s ago 24m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (9m) 0s ago 23m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:08:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (9m) 0s ago 23m 55.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (8m) 0s ago 22m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7m) 18s ago 22m 56.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6m) 18s ago 22m 53.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5m) 18s ago 21m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4m) 18s ago 21m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (13m) 18s ago 20m 65.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4m) 0s ago 18m 80.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4m) 18s ago 19m 78.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4m) 0s ago 18m 80.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:08:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4m) 18s ago 18m 79.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:08:29.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:08:29.008 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:08:29.275 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:08:29.276 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:08:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='client.55049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='client.45093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/507668275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='client.55049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='client.45093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:29.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/507668275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:08:29.588 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:08:29.588 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:08:29.588 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:08:29.588 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:08:29.588 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:08:29.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:29.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='client.55049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='client.45093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:29.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/507668275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='client.55061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='client.45111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1087502331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='client.55061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='client.45111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1087502331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='client.55061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='client.45111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1087502331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:30.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:31 smithi196 ceph-mon[76349]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:31 smithi184 ceph-mon[100622]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:31 smithi184 ceph-mon[102844]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:08:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:33 smithi184 ceph-mon[100622]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:33 smithi184 ceph-mon[102844]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:33 smithi196 ceph-mon[76349]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:33.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:33.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:33.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:35 smithi184 ceph-mon[100622]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:35 smithi184 ceph-mon[102844]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:35 smithi196 ceph-mon[76349]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:37 smithi196 ceph-mon[76349]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:37 smithi184 ceph-mon[100622]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:37 smithi184 ceph-mon[102844]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:39 smithi196 ceph-mon[76349]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:39 smithi184 ceph-mon[100622]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:39 smithi184 ceph-mon[102844]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:41 smithi196 ceph-mon[76349]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:41 smithi184 ceph-mon[100622]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:41 smithi184 ceph-mon[102844]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:08:43.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:43 smithi184 ceph-mon[100622]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:43.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:43 smithi184 ceph-mon[102844]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:43.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:43.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:43.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:43.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:43.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:43.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:43 smithi196 ceph-mon[76349]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:45 smithi184 ceph-mon[100622]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:45 smithi184 ceph-mon[102844]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:45 smithi196 ceph-mon[76349]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:47 smithi184 ceph-mon[100622]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:47 smithi184 ceph-mon[102844]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:47.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:47 smithi196 ceph-mon[76349]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[100622]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[102844]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:49 smithi196 ceph-mon[76349]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:51 smithi196 ceph-mon[76349]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[100622]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[102844]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:08:52.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:08:52] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:08:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:53 smithi196 ceph-mon[76349]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:53.489 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:53 smithi184 ceph-mon[100622]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:53.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:53 smithi184 ceph-mon[102844]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:53.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:53.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:08:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:08:53.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:08:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:55 smithi184 ceph-mon[102844]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:55 smithi184 ceph-mon[100622]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:55 smithi196 ceph-mon[76349]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:08:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:57 smithi184 ceph-mon[102844]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:57 smithi184 ceph-mon[100622]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:57 smithi196 ceph-mon[76349]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:08:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:59 smithi184 ceph-mon[100622]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:59 smithi184 ceph-mon[102844]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:59 smithi196 ceph-mon[76349]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:08:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:08:59.884 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 25m - - 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 25m - - 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (19m) 11s ago 21m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (20m) 10s ago 20m 91.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (13m) 11s ago 18m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (13m) 10s ago 24m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (14m) 11s ago 26m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:09:00.412 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (13m) 11s ago 26m 86.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (12m) 10s ago 25m 70.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (12m) 11s ago 25m 78.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (20m) 11s ago 20m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (20m) 10s ago 20m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (11m) 11s ago 24m 59.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (10m) 11s ago 24m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (9m) 11s ago 23m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (9m) 11s ago 23m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7m) 10s ago 22m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7m) 10s ago 22m 53.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5m) 10s ago 22m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5m) 10s ago 21m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (14m) 10s ago 21m 65.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:09:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4m) 11s ago 19m 80.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:09:00.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4m) 10s ago 19m 78.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:09:00.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4m) 11s ago 19m 80.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:09:00.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4m) 10s ago 19m 80.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:09:00.723 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:09:00.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:09:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:09:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:09:00.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:09:00.996 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:09:00.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:09:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[100622]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[100622]: from='client.45123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[100622]: from='client.45129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[100622]: from='client.45135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3151228697' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[102844]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[102844]: from='client.45123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[102844]: from='client.45129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[102844]: from='client.45135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3151228697' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:01.306 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:09:01.306 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:09:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:09:01.306 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:09:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:09:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:01 smithi196 ceph-mon[76349]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:01 smithi196 ceph-mon[76349]: from='client.45123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:01 smithi196 ceph-mon[76349]: from='client.45129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:01 smithi196 ceph-mon[76349]: from='client.45135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3151228697' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:02 smithi184 ceph-mon[100622]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1531216113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:02 smithi184 ceph-mon[102844]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1531216113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:02.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:09:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:02 smithi196 ceph-mon[76349]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:02.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1531216113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:03.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:03 smithi184 ceph-mon[100622]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:03 smithi184 ceph-mon[102844]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:03 smithi196 ceph-mon[76349]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:03.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:05 smithi184 ceph-mon[100622]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:05 smithi184 ceph-mon[102844]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:05 smithi196 ceph-mon[76349]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:07 smithi184 ceph-mon[100622]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:07 smithi184 ceph-mon[102844]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:07 smithi196 ceph-mon[76349]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[100622]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[102844]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:09 smithi196 ceph-mon[76349]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.456 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:11 smithi196 ceph-mon[76349]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:11.456 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.456 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[100622]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[102844]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:09:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:13 smithi196 ceph-mon[76349]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:13.691 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:13 smithi184 ceph-mon[100622]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:13.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:13 smithi184 ceph-mon[102844]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:13.691 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:13.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:13.692 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:13.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:13.692 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:13.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:14.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:14.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:15 smithi184 ceph-mon[100622]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:15 smithi184 ceph-mon[102844]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:15 smithi196 ceph-mon[76349]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:17 smithi196 ceph-mon[76349]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:17 smithi184 ceph-mon[100622]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:17 smithi184 ceph-mon[102844]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:19 smithi196 ceph-mon[76349]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:19 smithi184 ceph-mon[100622]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:19 smithi184 ceph-mon[102844]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:21 smithi196 ceph-mon[76349]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:21 smithi184 ceph-mon[100622]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:21 smithi184 ceph-mon[102844]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:22] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:09:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:23 smithi196 ceph-mon[76349]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:23 smithi184 ceph-mon[100622]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:23 smithi184 ceph-mon[102844]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:23.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:23.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:23.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:25 smithi196 ceph-mon[76349]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:25.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:25 smithi184 ceph-mon[100622]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:25.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:25 smithi184 ceph-mon[102844]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:27 smithi196 ceph-mon[76349]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:27 smithi184 ceph-mon[100622]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:27 smithi184 ceph-mon[102844]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:29 smithi196 ceph-mon[76349]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[102844]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[100622]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.605 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:09:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:31 smithi196 ceph-mon[76349]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:31.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[102844]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[100622]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 26m - - 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 26m - - 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (20m) 3s ago 21m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (21m) 1s ago 21m 91.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:09:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (14m) 3s ago 19m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (14m) 1s ago 25m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (15m) 3s ago 27m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (13m) 3s ago 27m 86.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (12m) 1s ago 25m 67.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (13m) 3s ago 25m 76.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (21m) 3s ago 21m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (21m) 1s ago 21m 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (11m) 3s ago 25m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (10m) 3s ago 24m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (10m) 3s ago 24m 56.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (9m) 3s ago 23m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (8m) 1s ago 23m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7m) 1s ago 23m 54.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:09:32.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6m) 1s ago 22m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5m) 1s ago 22m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (14m) 1s ago 21m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5m) 3s ago 20m 80.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5m) 1s ago 20m 78.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5m) 3s ago 19m 81.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:09:32.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5m) 1s ago 19m 80.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:09:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:09:32.458 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:09:32.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:09:32.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:09:32.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:09:32.459 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:09:32.737 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:09:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:09:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:09:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:09:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:09:32.739 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:09:33.062 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:09:33.062 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:09:33.062 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:09:33.062 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:09:33.062 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: from='client.55127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: from='client.55133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/6145700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1491065796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: from='client.55127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: from='client.55133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/6145700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1491065796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:33.641 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:33.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:33.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:33.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:33.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:33.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: from='client.55127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: from='client.55133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/6145700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:09:33.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1491065796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:09:34.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:34 smithi184 ceph-mon[100622]: from='client.55145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:34.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:34 smithi184 ceph-mon[102844]: from='client.55145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:34 smithi196 ceph-mon[76349]: from='client.55145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:35.628 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:35 smithi184 ceph-mon[100622]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:35.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:35 smithi184 ceph-mon[102844]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:35 smithi196 ceph-mon[76349]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:37 smithi196 ceph-mon[76349]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:37 smithi184 ceph-mon[100622]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:37 smithi184 ceph-mon[102844]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:39 smithi196 ceph-mon[76349]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:39 smithi184 ceph-mon[100622]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:39 smithi184 ceph-mon[102844]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:41 smithi196 ceph-mon[76349]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:41 smithi184 ceph-mon[100622]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:41 smithi184 ceph-mon[102844]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:09:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:43 smithi196 ceph-mon[76349]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:43 smithi184 ceph-mon[100622]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:43 smithi184 ceph-mon[102844]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:43.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:43.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:43.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:45 smithi184 ceph-mon[100622]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:45 smithi184 ceph-mon[102844]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:45 smithi196 ceph-mon[76349]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:47 smithi184 ceph-mon[102844]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:47 smithi184 ceph-mon[100622]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:47 smithi196 ceph-mon[76349]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[100622]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[102844]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:49 smithi196 ceph-mon[76349]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:50.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:50.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:51.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:51 smithi196 ceph-mon[76349]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:51.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[100622]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[102844]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:09:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:09:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:09:53.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:53 smithi196 ceph-mon[76349]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:53 smithi184 ceph-mon[100622]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:53 smithi184 ceph-mon[102844]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:53.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:53.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:09:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:09:53.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:09:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:55 smithi196 ceph-mon[76349]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:55 smithi184 ceph-mon[100622]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:55 smithi184 ceph-mon[102844]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:09:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:57 smithi184 ceph-mon[100622]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:57 smithi184 ceph-mon[102844]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:57 smithi196 ceph-mon[76349]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:09:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:59 smithi184 ceph-mon[102844]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:59 smithi184 ceph-mon[100622]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:09:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:59 smithi196 ceph-mon[76349]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:09:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:00.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:00.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:00.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[102844]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[102844]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[102844]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[102844]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[102844]: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[100622]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[100622]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[100622]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[100622]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:00 smithi184 ceph-mon[100622]: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:00 smithi196 ceph-mon[76349]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:00 smithi196 ceph-mon[76349]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:00 smithi196 ceph-mon[76349]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:00 smithi196 ceph-mon[76349]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:00 smithi196 ceph-mon[76349]: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:01 smithi184 ceph-mon[100622]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:01 smithi184 ceph-mon[102844]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:01 smithi196 ceph-mon[76349]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:02] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:10:03.360 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:10:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:03 smithi184 ceph-mon[100622]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:03 smithi184 ceph-mon[102844]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:03.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:03.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:03.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:03.890 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:10:03.890 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 26m - - 2024-09-20T20:10:03.890 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 26m - - 2024-09-20T20:10:03.890 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (20m) 15s ago 22m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (21m) 13s ago 21m 89.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (14m) 15s ago 20m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (14m) 13s ago 25m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (15m) 15s ago 27m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (14m) 15s ago 27m 87.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (13m) 13s ago 26m 68.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (13m) 15s ago 26m 76.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (21m) 15s ago 21m 14.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (21m) 13s ago 21m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (12m) 15s ago 25m 59.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (11m) 15s ago 25m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (10m) 15s ago 24m 56.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:10:03.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (10m) 15s ago 24m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (8m) 13s ago 24m 57.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (8m) 13s ago 23m 54.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7m) 13s ago 23m 57.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6m) 13s ago 22m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (15m) 13s ago 22m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6m) 15s ago 20m 81.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5m) 13s ago 20m 79.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5m) 15s ago 20m 81.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:10:03.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5m) 13s ago 20m 80.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:10:03.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:03 smithi196 ceph-mon[76349]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:04.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:10:04.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:10:04.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:10:04.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:10:04.203 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:10:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:10:04.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:10:04.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:04 smithi184 ceph-mon[102844]: from='client.55157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:04.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1558742635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:04.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:04 smithi184 ceph-mon[100622]: from='client.55157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:04.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1558742635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:04.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:04 smithi196 ceph-mon[76349]: from='client.55157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:04.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1558742635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:04.792 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:04.792 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:04.792 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:04.792 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:04.792 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:05 smithi196 ceph-mon[76349]: from='client.55163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:05 smithi196 ceph-mon[76349]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:05 smithi196 ceph-mon[76349]: from='client.55169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/683230109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[100622]: from='client.55163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[100622]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[100622]: from='client.55169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/683230109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[102844]: from='client.55163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[102844]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[102844]: from='client.55169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/683230109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:06 smithi184 ceph-mon[100622]: from='client.55181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:06 smithi184 ceph-mon[102844]: from='client.55181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:06 smithi196 ceph-mon[76349]: from='client.55181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:07 smithi184 ceph-mon[100622]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:07 smithi184 ceph-mon[102844]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:07 smithi196 ceph-mon[76349]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[100622]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[102844]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:09 smithi196 ceph-mon[76349]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:10.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:10.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:10.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[100622]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[102844]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:11 smithi196 ceph-mon[76349]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:12] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:10:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:12 smithi184 ceph-mon[100622]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:12 smithi184 ceph-mon[102844]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:12 smithi196 ceph-mon[76349]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:13.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:13.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:13.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:14.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:14.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:14 smithi184 ceph-mon[100622]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:14.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:14 smithi184 ceph-mon[102844]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:14 smithi196 ceph-mon[76349]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:16 smithi184 ceph-mon[100622]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:16 smithi184 ceph-mon[102844]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:16 smithi196 ceph-mon[76349]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:18 smithi184 ceph-mon[100622]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:18 smithi184 ceph-mon[102844]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:18 smithi196 ceph-mon[76349]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:20.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:20.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:20.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:20 smithi184 ceph-mon[100622]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:20 smithi184 ceph-mon[102844]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:20 smithi196 ceph-mon[76349]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:22] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:10:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:22 smithi184 ceph-mon[100622]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:22 smithi184 ceph-mon[102844]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:22 smithi196 ceph-mon[76349]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:23.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:23.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:23.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:25.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:24 smithi184 ceph-mon[100622]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:25.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:24 smithi184 ceph-mon[102844]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:25.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:24 smithi196 ceph-mon[76349]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:26 smithi184 ceph-mon[100622]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:26 smithi184 ceph-mon[102844]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:26 smithi196 ceph-mon[76349]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:28 smithi184 ceph-mon[100622]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:28 smithi184 ceph-mon[102844]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:28 smithi196 ceph-mon[76349]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:30.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:30.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:30.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:30.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:30.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[100622]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[102844]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:31 smithi196 ceph-mon[76349]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:32] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:10:33.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:33 smithi184 ceph-mon[100622]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:33.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:33 smithi184 ceph-mon[102844]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:33 smithi196 ceph-mon[76349]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:33.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:35.089 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:10:35.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:35 smithi184 ceph-mon[100622]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:35.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:35 smithi184 ceph-mon[102844]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:35.503 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:35 smithi196 ceph-mon[76349]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 27m - - 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 27m - - 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (21m) 6s ago 22m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (22m) 4s ago 22m 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (15m) 6s ago 20m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (15m) 4s ago 26m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (16m) 6s ago 28m 530M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:10:35.640 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (14m) 6s ago 28m 88.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (13m) 4s ago 26m 69.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (14m) 6s ago 26m 78.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (22m) 6s ago 22m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (22m) 4s ago 22m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (12m) 6s ago 26m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (11m) 6s ago 25m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (11m) 6s ago 25m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (10m) 6s ago 24m 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (9m) 4s ago 24m 57.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (8m) 4s ago 24m 54.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7m) 4s ago 23m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6m) 4s ago 23m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:10:35.641 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (15m) 4s ago 22m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:10:35.642 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6m) 6s ago 21m 81.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:10:35.642 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6m) 4s ago 21m 79.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:10:35.642 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6m) 6s ago 20m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:10:35.642 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6m) 4s ago 20m 81.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:10:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:10:35.956 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:10:36.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:10:36.230 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:10:36.230 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:10:36.230 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:10:36.231 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:10:36.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[100622]: from='client.45231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[100622]: from='client.45237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3116854293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:36.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[102844]: from='client.45231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[102844]: from='client.45237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3116854293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:36 smithi196 ceph-mon[76349]: from='client.45231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:36 smithi196 ceph-mon[76349]: from='client.45237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3116854293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:10:36.551 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:10:36.551 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:10:36.551 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[100622]: from='client.55205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[100622]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[100622]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2776236916' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[102844]: from='client.55205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[102844]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:37.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[102844]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2776236916' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:37 smithi196 ceph-mon[76349]: from='client.55205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:37 smithi196 ceph-mon[76349]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:37 smithi196 ceph-mon[76349]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2776236916' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:10:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:39 smithi184 ceph-mon[100622]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:39 smithi184 ceph-mon[102844]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:39 smithi196 ceph-mon[76349]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:40.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:40.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:40.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:41 smithi184 ceph-mon[100622]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:41 smithi184 ceph-mon[102844]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:41 smithi196 ceph-mon[76349]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:42] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:10:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:43 smithi184 ceph-mon[100622]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:43 smithi184 ceph-mon[102844]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:43 smithi196 ceph-mon[76349]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:43.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:43.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:43.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:45 smithi184 ceph-mon[100622]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:45 smithi184 ceph-mon[102844]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:45 smithi196 ceph-mon[76349]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:47 smithi184 ceph-mon[100622]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:47 smithi184 ceph-mon[102844]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:47 smithi196 ceph-mon[76349]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:49 smithi184 ceph-mon[100622]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:49 smithi184 ceph-mon[102844]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:49 smithi196 ceph-mon[76349]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:50.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:50.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:50.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:50.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[100622]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[102844]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:51 smithi196 ceph-mon[76349]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:10:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:10:52] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:53 smithi184 ceph-mon[100622]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:53 smithi184 ceph-mon[102844]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:53 smithi196 ceph-mon[76349]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:10:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:53.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:53.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:10:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:10:53.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:10:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:55 smithi196 ceph-mon[76349]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:55 smithi184 ceph-mon[100622]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:55 smithi184 ceph-mon[102844]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:10:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:57 smithi184 ceph-mon[100622]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:57 smithi184 ceph-mon[102844]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:57 smithi196 ceph-mon[76349]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:10:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:59 smithi184 ceph-mon[102844]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:10:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:59 smithi184 ceph-mon[100622]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:10:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:10:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:59 smithi196 ceph-mon[76349]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:10:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:10:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:00.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:00.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:00.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:01 smithi184 ceph-mon[100622]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:01 smithi184 ceph-mon[102844]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:01 smithi196 ceph-mon[76349]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:02] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:11:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:03 smithi184 ceph-mon[100622]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:03 smithi184 ceph-mon[102844]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:03 smithi196 ceph-mon[76349]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:03.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:03.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:03.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:05.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:05 smithi184 ceph-mon[100622]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:05.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:05 smithi184 ceph-mon[102844]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:05 smithi196 ceph-mon[76349]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:06.845 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 27m - - 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 27m - - 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (22m) 18s ago 23m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (22m) 16s ago 22m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (16m) 18s ago 21m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (15m) 16s ago 26m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (16m) 18s ago 28m 530M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (15m) 18s ago 28m 89.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (14m) 16s ago 27m 70.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (14m) 18s ago 27m 78.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (22m) 18s ago 22m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:11:07.372 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (22m) 16s ago 22m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (13m) 18s ago 26m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (12m) 18s ago 26m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (11m) 18s ago 25m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (11m) 18s ago 25m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (10m) 16s ago 25m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (9m) 16s ago 24m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (8m) 16s ago 24m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7m) 16s ago 23m 61.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (16m) 16s ago 23m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7m) 18s ago 21m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6m) 16s ago 21m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6m) 18s ago 21m 81.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:11:07.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6m) 16s ago 21m 81.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:11:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:07 smithi196 ceph-mon[76349]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:07.681 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:11:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:11:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:11:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:11:07.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:11:07.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:07 smithi184 ceph-mon[100622]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:07.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:07.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:07 smithi184 ceph-mon[102844]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:07.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:11:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:11:07.954 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:11:08.264 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:11:08.264 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:11:08.264 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:11:08.264 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:11:08.264 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:11:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[100622]: from='client.55229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[100622]: from='client.55235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[100622]: from='client.55241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4289513920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3231958289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[102844]: from='client.55229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[102844]: from='client.55235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[102844]: from='client.55241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4289513920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3231958289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:08 smithi196 ceph-mon[76349]: from='client.55229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:08 smithi196 ceph-mon[76349]: from='client.55235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:08 smithi196 ceph-mon[76349]: from='client.55241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4289513920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3231958289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:09 smithi196 ceph-mon[76349]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:09 smithi196 ceph-mon[76349]: from='client.55253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:09 smithi184 ceph-mon[102844]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:09 smithi184 ceph-mon[102844]: from='client.55253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:09 smithi184 ceph-mon[100622]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:09 smithi184 ceph-mon[100622]: from='client.55253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:10.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:10.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:10.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:10.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:10.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[100622]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[102844]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:11 smithi196 ceph-mon[76349]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:12] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:11:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:13 smithi184 ceph-mon[102844]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:13 smithi184 ceph-mon[100622]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:13.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:13.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:13.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:13 smithi196 ceph-mon[76349]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:14.685 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:15.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:15 smithi196 ceph-mon[76349]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:15 smithi184 ceph-mon[100622]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:15 smithi184 ceph-mon[102844]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:17.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:17 smithi184 ceph-mon[100622]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:17.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:17.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:17 smithi184 ceph-mon[102844]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:17.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:17 smithi196 ceph-mon[76349]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:19.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:19 smithi184 ceph-mon[100622]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:19.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:19 smithi184 ceph-mon[102844]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:19 smithi196 ceph-mon[76349]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:20.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:20.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:20.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:21 smithi184 ceph-mon[100622]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:21 smithi184 ceph-mon[102844]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:21 smithi196 ceph-mon[76349]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:22] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:11:23.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:23 smithi196 ceph-mon[76349]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:23 smithi184 ceph-mon[100622]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:23 smithi184 ceph-mon[102844]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:23.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:23.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:23.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:25 smithi184 ceph-mon[100622]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:25 smithi184 ceph-mon[102844]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:25 smithi196 ceph-mon[76349]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:27 smithi184 ceph-mon[100622]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:27 smithi184 ceph-mon[102844]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:27 smithi196 ceph-mon[76349]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:29 smithi184 ceph-mon[100622]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:29 smithi184 ceph-mon[102844]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:29 smithi196 ceph-mon[76349]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:30.529 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:30.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:30.529 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:30.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:30.529 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:30.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[102844]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:30 smithi184 ceph-mon[100622]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:30.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:30 smithi196 ceph-mon[76349]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:32] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:33 smithi184 ceph-mon[100622]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:33 smithi184 ceph-mon[102844]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:33 smithi196 ceph-mon[76349]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:33.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:33.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:33.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:35.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:35 smithi196 ceph-mon[76349]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:35 smithi184 ceph-mon[100622]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:35 smithi184 ceph-mon[102844]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:37.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:37 smithi196 ceph-mon[76349]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:37.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:37 smithi184 ceph-mon[100622]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:37 smithi184 ceph-mon[102844]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:11:39.116 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 28m - - 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 28m - - 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (22m) 9s ago 23m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (23m) 7s ago 23m 88.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (16m) 9s ago 21m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (16m) 7s ago 27m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (17m) 9s ago 29m 531M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (15m) 9s ago 29m 88.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (15m) 7s ago 27m 71.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (15m) 9s ago 27m 77.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (23m) 9s ago 23m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (23m) 7s ago 23m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:11:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (13m) 9s ago 27m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (13m) 9s ago 26m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (12m) 9s ago 26m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (11m) 9s ago 26m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (10m) 7s ago 25m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (9m) 7s ago 25m 54.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (8m) 7s ago 24m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7m) 7s ago 24m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (16m) 7s ago 24m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7m) 9s ago 22m 82.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7m) 7s ago 22m 80.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7m) 9s ago 21m 82.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:11:39.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7m) 7s ago 21m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:11:39.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:39 smithi184 ceph-mon[100622]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:39.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:39 smithi184 ceph-mon[102844]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:11:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:11:39.450 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:11:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:39 smithi196 ceph-mon[76349]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:39.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:11:39.727 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:11:39.728 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:11:39.729 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:11:39.729 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:11:39.729 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:11:39.729 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:11:39.729 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:11:40.044 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:11:40.044 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:11:40.044 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:11:40.044 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:11:40.044 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:11:40.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:40.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:40.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:40.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:40.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:40.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[102844]: from='client.45303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[102844]: from='client.55271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[102844]: from='client.45315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4160230882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/77072173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[100622]: from='client.45303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[100622]: from='client.55271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[100622]: from='client.45315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4160230882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/77072173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:40 smithi196 ceph-mon[76349]: from='client.45303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:40 smithi196 ceph-mon[76349]: from='client.55271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:40 smithi196 ceph-mon[76349]: from='client.45315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4160230882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:11:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/77072173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:11:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:41 smithi184 ceph-mon[100622]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:41 smithi184 ceph-mon[100622]: from='client.45327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:41 smithi184 ceph-mon[102844]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:41 smithi184 ceph-mon[102844]: from='client.45327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:41 smithi196 ceph-mon[76349]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:41 smithi196 ceph-mon[76349]: from='client.45327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:42] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:11:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:43 smithi184 ceph-mon[100622]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:43 smithi184 ceph-mon[102844]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:43 smithi196 ceph-mon[76349]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:43.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:43.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:43.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:45 smithi184 ceph-mon[100622]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:45 smithi184 ceph-mon[102844]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:45 smithi196 ceph-mon[76349]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:47 smithi184 ceph-mon[100622]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:47 smithi184 ceph-mon[102844]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:47 smithi196 ceph-mon[76349]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:49 smithi184 ceph-mon[100622]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:49 smithi184 ceph-mon[102844]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:49 smithi196 ceph-mon[76349]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:50.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:50.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:50.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[100622]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:50 smithi184 ceph-mon[102844]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:51.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:50 smithi196 ceph-mon[76349]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:11:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:11:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:53 smithi196 ceph-mon[76349]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:53 smithi184 ceph-mon[100622]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:53 smithi184 ceph-mon[102844]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:53.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:53.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:11:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:11:53.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:55 smithi196 ceph-mon[76349]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:55 smithi184 ceph-mon[100622]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:55 smithi184 ceph-mon[102844]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:11:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:57 smithi196 ceph-mon[76349]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:57 smithi184 ceph-mon[100622]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:57 smithi184 ceph-mon[102844]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:11:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:59 smithi184 ceph-mon[100622]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:59 smithi184 ceph-mon[102844]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:59 smithi196 ceph-mon[76349]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:00.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:00.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:00.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:00.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:00.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:00.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:01 smithi184 ceph-mon[100622]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:01 smithi184 ceph-mon[102844]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:01 smithi196 ceph-mon[76349]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:12:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:03 smithi196 ceph-mon[76349]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:03 smithi184 ceph-mon[100622]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:03 smithi184 ceph-mon[102844]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:03.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:03.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:03.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:05 smithi196 ceph-mon[76349]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:05 smithi184 ceph-mon[100622]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:05 smithi184 ceph-mon[102844]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:07 smithi184 ceph-mon[100622]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:07 smithi184 ceph-mon[102844]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:07 smithi196 ceph-mon[76349]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:09 smithi184 ceph-mon[100622]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:09 smithi184 ceph-mon[102844]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:09 smithi196 ceph-mon[76349]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:12:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:10.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:10.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:10.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:12:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 28m - - 2024-09-20T20:12:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 28m - - 2024-09-20T20:12:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (23m) 1s ago 24m 30.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (23m) 19s ago 23m 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (17m) 1s ago 22m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (16m) 19s ago 28m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (17m) 1s ago 29m 531M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (16m) 1s ago 29m 89.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (15m) 19s ago 28m 69.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (15m) 1s ago 28m 78.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (23m) 1s ago 23m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (23m) 19s ago 23m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (14m) 1s ago 27m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (13m) 1s ago 27m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (12m) 1s ago 26m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (12m) 1s ago 26m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (11m) 19s ago 26m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:12:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (10m) 19s ago 25m 55.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (9m) 19s ago 25m 58.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (8m) 19s ago 25m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (17m) 19s ago 24m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (8m) 1s ago 22m 82.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7m) 19s ago 22m 80.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (8m) 1s ago 22m 82.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:12:10.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7m) 19s ago 22m 81.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:12:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[100622]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[100622]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[102844]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:10 smithi184 ceph-mon[102844]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:10 smithi196 ceph-mon[76349]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:11.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:10 smithi196 ceph-mon[76349]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:11.168 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:12:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:11.170 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:12:11.170 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:12:11.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:12:11.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:12:11.438 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:12:11.438 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:12:11.439 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:12:11.755 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:12:11.755 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:12:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:12:11.755 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:12:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:12:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='client.55307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='client.45351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/441758538' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='client.55325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3890951266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='client.55307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='client.45351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/441758538' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:12.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='client.55325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3890951266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='client.55307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='client.45351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/441758538' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='client.55325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:12.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3890951266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:12 smithi184 ceph-mon[100622]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:12 smithi184 ceph-mon[102844]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:12 smithi196 ceph-mon[76349]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:13.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:13.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:13.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:14 smithi184 ceph-mon[100622]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:14 smithi184 ceph-mon[102844]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:14 smithi196 ceph-mon[76349]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:17.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:16 smithi196 ceph-mon[76349]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:17.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:16 smithi184 ceph-mon[100622]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:16 smithi184 ceph-mon[102844]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:18 smithi196 ceph-mon[76349]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:19.085 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:18 smithi184 ceph-mon[100622]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:19.085 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:18 smithi184 ceph-mon[102844]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:20.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:20.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:20.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:20.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:20 smithi184 ceph-mon[100622]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:20.941 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:20 smithi184 ceph-mon[102844]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:20 smithi196 ceph-mon[76349]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:22] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:12:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:22 smithi196 ceph-mon[76349]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:22 smithi184 ceph-mon[102844]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:22 smithi184 ceph-mon[100622]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:23.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:23.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:23.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:24 smithi196 ceph-mon[76349]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:24 smithi184 ceph-mon[100622]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:24 smithi184 ceph-mon[102844]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:26 smithi184 ceph-mon[100622]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:26 smithi184 ceph-mon[102844]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:26 smithi196 ceph-mon[76349]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:28 smithi196 ceph-mon[76349]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:28 smithi184 ceph-mon[100622]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:28 smithi184 ceph-mon[102844]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:30.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:30.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:30.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[100622]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[102844]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:30 smithi196 ceph-mon[76349]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:12:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:32 smithi196 ceph-mon[76349]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[100622]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:32 smithi184 ceph-mon[102844]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:33.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:33.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:33.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:34 smithi184 ceph-mon[100622]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:34 smithi184 ceph-mon[102844]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:34 smithi196 ceph-mon[76349]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:36 smithi184 ceph-mon[100622]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:36 smithi184 ceph-mon[102844]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:37.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:36 smithi196 ceph-mon[76349]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:37.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:38 smithi184 ceph-mon[100622]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:38 smithi184 ceph-mon[102844]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:38 smithi196 ceph-mon[76349]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:40.413 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:40.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:40.413 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:40.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:40.413 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:40.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:40 smithi184 ceph-mon[100622]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:40 smithi184 ceph-mon[102844]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:41.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:40 smithi196 ceph-mon[76349]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:42.043 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:12:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:12:42.570 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:12:42.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 29m - - 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 29m - - 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (23m) 12s ago 24m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (24m) 10s ago 24m 91.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (17m) 12s ago 22m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (17m) 10s ago 28m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (18m) 12s ago 30m 531M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (16m) 12s ago 30m 90.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (16m) 10s ago 29m 70.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (16m) 12s ago 28m 78.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (24m) 12s ago 24m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (24m) 10s ago 24m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (14m) 12s ago 28m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (14m) 12s ago 27m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (13m) 12s ago 27m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:12:42.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (12m) 12s ago 27m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (11m) 10s ago 26m 58.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (10m) 10s ago 26m 55.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (9m) 10s ago 25m 58.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (9m) 10s ago 25m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (17m) 10s ago 25m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (8m) 12s ago 23m 82.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (8m) 10s ago 23m 80.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (8m) 12s ago 22m 82.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:12:42.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (8m) 10s ago 22m 82.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:12:42.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:12:42.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:12:42.884 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[100622]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:42.884 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[100622]: from='client.55337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:42.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[100622]: from='client.45381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:42.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[102844]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:42.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[102844]: from='client.55337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:42.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:42 smithi184 ceph-mon[102844]: from='client.45381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:42 smithi196 ceph-mon[76349]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:42 smithi196 ceph-mon[76349]: from='client.55337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:42 smithi196 ceph-mon[76349]: from='client.45381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:43.151 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:12:43.152 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:12:43.464 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:12:43.464 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:12:43.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:12:43.464 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:12:43.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:12:43.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:43.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:43.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:43.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:43.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:43.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[100622]: from='client.45387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2864411469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[100622]: from='client.55361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/159757261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:43.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[102844]: from='client.45387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2864411469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[102844]: from='client.55361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/159757261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:43 smithi196 ceph-mon[76349]: from='client.45387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2864411469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:12:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:43 smithi196 ceph-mon[76349]: from='client.55361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/159757261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:12:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:44 smithi184 ceph-mon[100622]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:44 smithi184 ceph-mon[102844]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:44 smithi196 ceph-mon[76349]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:46 smithi196 ceph-mon[76349]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:46 smithi184 ceph-mon[100622]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:46 smithi184 ceph-mon[102844]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:48 smithi196 ceph-mon[76349]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:48 smithi184 ceph-mon[100622]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:48 smithi184 ceph-mon[102844]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:50.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:50.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:50.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:51.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[100622]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:51.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[102844]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:51.334 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.334 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.334 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:51 smithi196 ceph-mon[76349]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:12:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[100622]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[102844]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:52 smithi196 ceph-mon[76349]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:12:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:53.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:53.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:12:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:12:53.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:12:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:54 smithi184 ceph-mon[100622]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:54 smithi184 ceph-mon[102844]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:54 smithi196 ceph-mon[76349]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:12:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:56 smithi184 ceph-mon[100622]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:56 smithi184 ceph-mon[102844]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:56 smithi196 ceph-mon[76349]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:12:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:58 smithi196 ceph-mon[76349]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:12:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:58 smithi184 ceph-mon[100622]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:12:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:12:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:58 smithi184 ceph-mon[102844]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:12:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:12:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:00.212 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:00.157Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:00.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:00.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:00.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:00.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:00 smithi184 ceph-mon[100622]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:00 smithi184 ceph-mon[102844]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:00 smithi196 ceph-mon[76349]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:02.283 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:13:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:02 smithi184 ceph-mon[102844]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:02 smithi184 ceph-mon[100622]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:02 smithi196 ceph-mon[76349]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:03.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:03.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:03.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:04 smithi184 ceph-mon[102844]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:04 smithi184 ceph-mon[100622]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:04 smithi196 ceph-mon[76349]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:06 smithi184 ceph-mon[100622]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:06 smithi184 ceph-mon[102844]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:06 smithi196 ceph-mon[76349]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:08 smithi184 ceph-mon[100622]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:08 smithi184 ceph-mon[102844]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:08 smithi196 ceph-mon[76349]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:10.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:10.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:10.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[100622]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[102844]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:11 smithi196 ceph-mon[76349]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:13:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:12 smithi196 ceph-mon[76349]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[102844]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[100622]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:13.764 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:13:13.871 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:13.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:13.871 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:13.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:13.871 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:13.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:14.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:14.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 29m - - 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 29m - - 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (24m) 4s ago 25m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (25m) 2s ago 25m 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (18m) 4s ago 23m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (17m) 2s ago 29m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:13:14.295 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (18m) 4s ago 30m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (17m) 4s ago 30m 92.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (16m) 2s ago 29m 72.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (16m) 4s ago 29m 80.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (24m) 4s ago 24m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (24m) 2s ago 24m 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (15m) 4s ago 28m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (14m) 4s ago 28m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (13m) 4s ago 27m 56.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (13m) 4s ago 27m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (12m) 2s ago 27m 58.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (11m) 2s ago 26m 55.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (10m) 2s ago 26m 58.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (9m) 2s ago 26m 62.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:13:14.296 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (18m) 2s ago 25m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:13:14.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (9m) 4s ago 23m 83.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:13:14.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (8m) 2s ago 23m 81.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:13:14.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (9m) 4s ago 23m 83.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:13:14.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (8m) 2s ago 23m 82.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:13:14.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:13:14.881 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:13:14.882 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:13:14.882 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:13:14.882 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:13:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:14 smithi196 ceph-mon[76349]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:14 smithi196 ceph-mon[76349]: from='client.55373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:14 smithi196 ceph-mon[76349]: from='client.45417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:14 smithi196 ceph-mon[76349]: from='client.55385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/718287314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:15.189 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:13:15.189 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:13:15.189 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:13:15.189 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:13:15.189 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[100622]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[100622]: from='client.55373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[100622]: from='client.45417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[100622]: from='client.55385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/718287314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[102844]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[102844]: from='client.55373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[102844]: from='client.45417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[102844]: from='client.55385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:15.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/718287314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:15 smithi196 ceph-mon[76349]: from='client.55397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1340971811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:15 smithi184 ceph-mon[100622]: from='client.55397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1340971811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:15 smithi184 ceph-mon[102844]: from='client.55397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1340971811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:17.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:16 smithi196 ceph-mon[76349]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:17.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:16 smithi184 ceph-mon[100622]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:16 smithi184 ceph-mon[102844]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:18 smithi196 ceph-mon[76349]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:18 smithi184 ceph-mon[100622]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:18 smithi184 ceph-mon[102844]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:20.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:20.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:20.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:21.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:20 smithi196 ceph-mon[76349]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:21.095 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:20 smithi184 ceph-mon[100622]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:21.095 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:20 smithi184 ceph-mon[102844]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:22.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:22] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:13:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:22 smithi196 ceph-mon[76349]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:22 smithi184 ceph-mon[100622]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:22 smithi184 ceph-mon[102844]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:23.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:23.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:23.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:23.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:23.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:23.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:24 smithi196 ceph-mon[76349]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:24 smithi184 ceph-mon[100622]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:24 smithi184 ceph-mon[102844]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:26 smithi184 ceph-mon[100622]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:26 smithi184 ceph-mon[102844]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:26 smithi196 ceph-mon[76349]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:28 smithi184 ceph-mon[102844]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:28 smithi184 ceph-mon[100622]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:28 smithi196 ceph-mon[76349]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:30.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:30.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:30.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:31 smithi196 ceph-mon[76349]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[100622]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[102844]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:32] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[102844]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[100622]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:32 smithi196 ceph-mon[76349]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:33.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:33.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:33.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:33.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:34 smithi184 ceph-mon[100622]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:34 smithi184 ceph-mon[102844]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:34 smithi196 ceph-mon[76349]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:37 smithi184 ceph-mon[100622]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:37 smithi184 ceph-mon[102844]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:37 smithi196 ceph-mon[76349]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:39.095 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:39 smithi196 ceph-mon[76349]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:39 smithi184 ceph-mon[100622]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:39.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:39 smithi184 ceph-mon[102844]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:40.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:40.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:40.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:41.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:41 smithi196 ceph-mon[76349]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:41 smithi184 ceph-mon[100622]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:41 smithi184 ceph-mon[102844]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:42.229 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:42] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:13:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:43 smithi184 ceph-mon[100622]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:43 smithi184 ceph-mon[102844]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:43 smithi196 ceph-mon[76349]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:43.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:43.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:43.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:43.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:44.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:44.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:45 smithi184 ceph-mon[100622]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:45 smithi184 ceph-mon[102844]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:45 smithi196 ceph-mon[76349]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:45.496 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 30m - - 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 30m - - 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (24m) 15s ago 25m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (25m) 14s ago 25m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (18m) 15s ago 23m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (18m) 14s ago 29m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (19m) 15s ago 31m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (17m) 15s ago 31m 93.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (17m) 14s ago 30m 72.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (17m) 15s ago 29m 81.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:13:46.025 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (25m) 15s ago 25m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (25m) 14s ago 25m 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (16m) 15s ago 29m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (15m) 15s ago 28m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (14m) 15s ago 28m 56.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (14m) 15s ago 28m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (12m) 14s ago 27m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (11m) 14s ago 27m 55.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (10m) 14s ago 27m 58.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (10m) 14s ago 26m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (18m) 14s ago 26m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (9m) 15s ago 24m 83.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (9m) 14s ago 24m 81.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (9m) 15s ago 23m 83.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:13:46.026 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (9m) 14s ago 23m 82.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:13:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:46 smithi184 ceph-mon[100622]: from='client.45447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:46 smithi184 ceph-mon[102844]: from='client.45447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:13:46.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:13:46.341 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:13:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:46 smithi196 ceph-mon[76349]: from='client.45447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:46.611 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:13:46.611 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:13:46.612 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:13:46.923 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:13:46.923 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:13:46.923 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:13:46.923 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:13:46.923 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:13:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: from='client.45453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: from='client.55421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/818732277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3636554132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: from='client.45453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: from='client.55421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:47.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/818732277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:47.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3636554132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: from='client.45453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: from='client.55421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/818732277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:13:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3636554132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:13:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:48 smithi184 ceph-mon[100622]: from='client.55433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:48 smithi184 ceph-mon[102844]: from='client.55433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:48 smithi196 ceph-mon[76349]: from='client.55433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:49 smithi184 ceph-mon[100622]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:49 smithi184 ceph-mon[102844]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:49 smithi196 ceph-mon[76349]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:50.454 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:50.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:50.454 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:50.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:50.454 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:50.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[100622]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[102844]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:51 smithi196 ceph-mon[76349]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:13:52] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:13:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:53 smithi196 ceph-mon[76349]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[100622]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[102844]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:13:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:53.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:53.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:13:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:13:53.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:13:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:55 smithi196 ceph-mon[76349]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:55 smithi184 ceph-mon[100622]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:55 smithi184 ceph-mon[102844]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:13:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:57 smithi196 ceph-mon[76349]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:57 smithi184 ceph-mon[100622]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:57 smithi184 ceph-mon[102844]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:59 smithi196 ceph-mon[76349]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:59 smithi184 ceph-mon[100622]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:13:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:59 smithi184 ceph-mon[102844]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:13:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:00.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:00.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:00.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:01.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:01 smithi196 ceph-mon[76349]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:01 smithi184 ceph-mon[100622]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:01 smithi184 ceph-mon[102844]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:02] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:14:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:03 smithi196 ceph-mon[76349]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:03.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:03 smithi184 ceph-mon[100622]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:03.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:03 smithi184 ceph-mon[102844]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:03.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:03.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:03.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:03.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:05 smithi184 ceph-mon[100622]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:05 smithi184 ceph-mon[102844]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:14:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:05 smithi196 ceph-mon[76349]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:14:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:07 smithi184 ceph-mon[100622]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:07 smithi184 ceph-mon[102844]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:07 smithi196 ceph-mon[76349]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:09 smithi184 ceph-mon[100622]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:14:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:09 smithi184 ceph-mon[102844]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:14:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:09 smithi196 ceph-mon[76349]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:14:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:10.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:10.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:10.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:11 smithi196 ceph-mon[76349]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[100622]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[102844]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:12] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:14:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[102844]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[100622]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:13 smithi196 ceph-mon[76349]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:14:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:13.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:13.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:13.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:15 smithi184 ceph-mon[100622]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:14:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:15 smithi184 ceph-mon[102844]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:14:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:15 smithi196 ceph-mon[76349]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:14:17.248 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:14:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:17 smithi184 ceph-mon[102844]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:17 smithi184 ceph-mon[100622]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:17 smithi196 ceph-mon[76349]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 31m - - 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 30m - - 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (25m) 7s ago 26m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (26m) 5s ago 26m 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:14:17.776 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (19m) 7s ago 24m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (19m) 5s ago 30m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (19m) 7s ago 31m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (18m) 7s ago 31m 94.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (17m) 5s ago 30m 74.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (18m) 7s ago 30m 82.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (25m) 7s ago 25m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (25m) 5s ago 25m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (16m) 7s ago 29m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (15m) 7s ago 29m 61.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (15m) 7s ago 29m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (14m) 7s ago 28m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (13m) 5s ago 28m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (12m) 5s ago 27m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (11m) 5s ago 27m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:14:17.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (10m) 5s ago 27m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:14:17.778 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (19m) 5s ago 26m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:14:17.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (10m) 7s ago 24m 83.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:14:17.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (10m) 5s ago 24m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:14:17.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (10m) 7s ago 24m 83.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:14:17.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (9m) 5s ago 24m 83.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:14:18.087 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:14:18.087 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:14:18.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:14:18.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:14:18.089 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:14:18.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:18 smithi184 ceph-mon[100622]: from='client.55445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:18.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/22504531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:18.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:18 smithi184 ceph-mon[102844]: from='client.55445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:18.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/22504531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:18.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:14:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:14:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:14:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:14:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:14:18.362 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:14:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:18 smithi196 ceph-mon[76349]: from='client.55445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/22504531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:18.672 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:14:18.672 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:14:18.672 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:14:18.672 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:14:18.672 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:14:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[100622]: from='client.55451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[100622]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[100622]: from='client.55457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[100622]: from='client.55469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3071156376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[102844]: from='client.55451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[102844]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[102844]: from='client.55457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[102844]: from='client.55469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3071156376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:19 smithi196 ceph-mon[76349]: from='client.55451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:19 smithi196 ceph-mon[76349]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:14:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:19 smithi196 ceph-mon[76349]: from='client.55457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:19 smithi196 ceph-mon[76349]: from='client.55469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3071156376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:20.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:20.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:20.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:21 smithi184 ceph-mon[100622]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T20:14:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:21 smithi184 ceph-mon[102844]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T20:14:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:21 smithi196 ceph-mon[76349]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T20:14:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:22] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:14:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:23 smithi184 ceph-mon[100622]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:14:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:23 smithi184 ceph-mon[102844]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:14:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:23 smithi196 ceph-mon[76349]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:14:23.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:23.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:23.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:23.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:23.968 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:23.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:25.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:25 smithi184 ceph-mon[100622]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:14:25.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:25 smithi184 ceph-mon[102844]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:25 smithi196 ceph-mon[76349]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:14:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:27 smithi184 ceph-mon[102844]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:14:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:27 smithi184 ceph-mon[100622]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:14:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:27 smithi196 ceph-mon[76349]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:14:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:29 smithi184 ceph-mon[102844]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T20:14:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:29 smithi184 ceph-mon[100622]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T20:14:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:29 smithi196 ceph-mon[76349]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T20:14:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:30.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:30.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:30.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[102844]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[100622]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:31 smithi196 ceph-mon[76349]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:14:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:32] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:14:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:33 smithi196 ceph-mon[76349]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:14:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[102844]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[100622]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:33.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:33.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:33.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:35 smithi196 ceph-mon[76349]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:35 smithi184 ceph-mon[102844]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:35 smithi184 ceph-mon[100622]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:14:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:37 smithi184 ceph-mon[102844]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T20:14:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:37 smithi184 ceph-mon[100622]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T20:14:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:37 smithi196 ceph-mon[76349]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T20:14:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:39 smithi184 ceph-mon[102844]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:14:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:39 smithi184 ceph-mon[100622]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:14:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:39 smithi196 ceph-mon[76349]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:14:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:40.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:40.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:40.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:41 smithi196 ceph-mon[76349]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:14:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:41 smithi184 ceph-mon[100622]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:14:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:41 smithi184 ceph-mon[102844]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:14:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:42] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:14:43.687 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:43 smithi184 ceph-mon[100622]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:43.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:43 smithi184 ceph-mon[102844]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:43.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:43.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:43.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:43.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:43.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:43.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:43 smithi196 ceph-mon[76349]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:44.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:44.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:45 smithi184 ceph-mon[100622]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:45 smithi184 ceph-mon[102844]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:45 smithi196 ceph-mon[76349]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:47 smithi184 ceph-mon[102844]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:47 smithi184 ceph-mon[100622]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:47 smithi196 ceph-mon[76349]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:48.957 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:14:49.486 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 31m - - 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 31m - - 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (25m) 18s ago 26m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (26m) 17s ago 26m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (19m) 18s ago 24m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (19m) 17s ago 30m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (20m) 18s ago 32m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (18m) 18s ago 32m 95.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (18m) 17s ago 31m 74.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (18m) 18s ago 30m 83.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (26m) 18s ago 26m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:14:49.487 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (26m) 17s ago 26m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (17m) 18s ago 30m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (16m) 18s ago 29m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (15m) 18s ago 29m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (15m) 18s ago 29m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (13m) 17s ago 28m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (12m) 17s ago 28m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (11m) 17s ago 28m 60.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (11m) 17s ago 27m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (19m) 17s ago 27m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (10m) 18s ago 25m 83.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (10m) 17s ago 25m 81.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (10m) 18s ago 24m 84.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:14:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (10m) 17s ago 24m 83.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:14:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:49 smithi184 ceph-mon[100622]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:49 smithi184 ceph-mon[102844]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:14:49.800 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:14:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:49 smithi196 ceph-mon[76349]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:14:50.069 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:14:50.070 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:14:50.070 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:14:50.070 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:14:50.070 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:14:50.070 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:14:50.377 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:14:50.378 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:14:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:14:50.378 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:14:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:14:50.378 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:50.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:50.378 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:50.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:50.378 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:50.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[100622]: from='client.55481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[100622]: from='client.55487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[100622]: from='client.55493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3602661279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3259693597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[102844]: from='client.55481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[102844]: from='client.55487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[102844]: from='client.55493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3602661279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3259693597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:50 smithi196 ceph-mon[76349]: from='client.55481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:50 smithi196 ceph-mon[76349]: from='client.55487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:50 smithi196 ceph-mon[76349]: from='client.55493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3602661279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3259693597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[100622]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[100622]: from='client.45543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[102844]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[102844]: from='client.45543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:51 smithi196 ceph-mon[76349]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:51 smithi196 ceph-mon[76349]: from='client.45543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:14:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:14:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[100622]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[102844]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:53.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:53.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:14:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:14:53.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:14:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:53 smithi196 ceph-mon[76349]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:14:55.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:55 smithi196 ceph-mon[76349]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:55 smithi184 ceph-mon[100622]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:55 smithi184 ceph-mon[102844]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:14:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:57 smithi184 ceph-mon[100622]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:57 smithi184 ceph-mon[102844]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:57 smithi196 ceph-mon[76349]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:14:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:59 smithi184 ceph-mon[100622]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:59 smithi184 ceph-mon[102844]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:14:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:59 smithi196 ceph-mon[76349]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:14:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:00.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:00.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:00.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:01 smithi184 ceph-mon[100622]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:01 smithi184 ceph-mon[102844]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:01 smithi196 ceph-mon[76349]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:02] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:15:03.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:03 smithi196 ceph-mon[76349]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:03 smithi184 ceph-mon[102844]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:03 smithi184 ceph-mon[100622]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:03.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:03.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:03.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:05.743 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:05 smithi184 ceph-mon[100622]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:05.743 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:05 smithi184 ceph-mon[102844]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:05 smithi196 ceph-mon[76349]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:07 smithi184 ceph-mon[100622]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:07 smithi184 ceph-mon[102844]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:07 smithi196 ceph-mon[76349]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:09 smithi184 ceph-mon[102844]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:09 smithi184 ceph-mon[100622]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:09 smithi196 ceph-mon[76349]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:15:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:10.168Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:10.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:10.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[100622]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[102844]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:11 smithi196 ceph-mon[76349]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:12] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[102844]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[100622]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:13.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:13.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:13.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:13 smithi196 ceph-mon[76349]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:15 smithi184 ceph-mon[100622]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:15 smithi184 ceph-mon[102844]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:15 smithi196 ceph-mon[76349]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:17 smithi196 ceph-mon[76349]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:17 smithi184 ceph-mon[100622]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:17 smithi184 ceph-mon[102844]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:19 smithi184 ceph-mon[100622]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:19 smithi184 ceph-mon[102844]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:19 smithi196 ceph-mon[76349]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:20.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:20.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:20.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:20.675 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:15:21.202 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 32m - - 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 31m - - 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (26m) 10s ago 27m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (27m) 8s ago 27m 91.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (20m) 10s ago 25m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (20m) 8s ago 31m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (20m) 10s ago 32m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (19m) 10s ago 33m 96.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (18m) 8s ago 31m 76.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (19m) 10s ago 31m 84.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (26m) 10s ago 26m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (26m) 8s ago 26m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (17m) 10s ago 30m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:15:21.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (16m) 10s ago 30m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (16m) 10s ago 30m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (15m) 10s ago 29m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (14m) 8s ago 29m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (13m) 8s ago 28m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (12m) 8s ago 28m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (11m) 8s ago 28m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (20m) 8s ago 27m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (11m) 10s ago 25m 84.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (11m) 8s ago 25m 82.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (11m) 10s ago 25m 84.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:15:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (10m) 8s ago 25m 83.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:15:21.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:15:21.519 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:15:21.787 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:15:21.788 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:15:21.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:21 smithi184 ceph-mon[100622]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:21.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:21 smithi184 ceph-mon[102844]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:21 smithi196 ceph-mon[76349]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:22] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:15:22.100 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:15:22.100 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:15:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:15:22.100 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:15:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[100622]: from='client.45555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[100622]: from='client.55523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[100622]: from='client.55529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/650103962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/187285812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[102844]: from='client.45555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[102844]: from='client.55523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[102844]: from='client.55529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/650103962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/187285812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:22 smithi196 ceph-mon[76349]: from='client.45555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:22 smithi196 ceph-mon[76349]: from='client.55523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:22 smithi196 ceph-mon[76349]: from='client.55529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/650103962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/187285812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:23 smithi196 ceph-mon[76349]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:23 smithi196 ceph-mon[76349]: from='client.55541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-mon[102844]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-mon[102844]: from='client.55541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-mon[100622]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-mon[100622]: from='client.55541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:23.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:23.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:23.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:25 smithi184 ceph-mon[100622]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:25 smithi184 ceph-mon[102844]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:25 smithi196 ceph-mon[76349]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:27 smithi184 ceph-mon[100622]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:27 smithi184 ceph-mon[102844]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:27 smithi196 ceph-mon[76349]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:29 smithi184 ceph-mon[100622]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:29 smithi184 ceph-mon[102844]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:29 smithi196 ceph-mon[76349]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:30.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:30.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:30.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[100622]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[102844]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:31 smithi196 ceph-mon[76349]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:32] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[100622]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[102844]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:33.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:33.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:33.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:33 smithi196 ceph-mon[76349]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:34 smithi196 ceph-mon[76349]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:34 smithi184 ceph-mon[102844]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:34 smithi184 ceph-mon[100622]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:36 smithi196 ceph-mon[76349]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:36 smithi184 ceph-mon[102844]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:36 smithi184 ceph-mon[100622]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:38 smithi196 ceph-mon[76349]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:38 smithi184 ceph-mon[100622]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:38 smithi184 ceph-mon[102844]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:40.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:40.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:40.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:40 smithi196 ceph-mon[76349]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:40 smithi184 ceph-mon[100622]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:40 smithi184 ceph-mon[102844]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:42] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:15:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:42 smithi196 ceph-mon[76349]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:42 smithi184 ceph-mon[100622]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:42 smithi184 ceph-mon[102844]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:43.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:43.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:43.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:43.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:43.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:43.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:44 smithi196 ceph-mon[76349]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:44 smithi184 ceph-mon[100622]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:44 smithi184 ceph-mon[102844]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:47.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:46 smithi184 ceph-mon[100622]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:47.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:47.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:46 smithi184 ceph-mon[102844]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:47.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:46 smithi196 ceph-mon[76349]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:49.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:48 smithi184 ceph-mon[100622]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:49.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:48 smithi184 ceph-mon[102844]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:48 smithi196 ceph-mon[76349]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:50.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:50.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:50.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:50 smithi196 ceph-mon[76349]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:51.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:50 smithi184 ceph-mon[100622]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:51.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:50 smithi184 ceph-mon[102844]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:15:52] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:15:52.401 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:15:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:52.939 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 32m - - 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 32m - - 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (26m) 1s ago 27m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (27m) 0s ago 27m 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (20m) 1s ago 25m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (20m) 0s ago 31m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (21m) 1s ago 33m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (19m) 1s ago 33m 96.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (19m) 0s ago 32m 74.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (19m) 1s ago 32m 82.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (27m) 1s ago 27m 15.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:15:52.940 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (27m) 0s ago 27m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (18m) 1s ago 31m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (17m) 1s ago 31m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (16m) 1s ago 30m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (16m) 1s ago 30m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (14m) 0s ago 29m 61.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (13m) 0s ago 29m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (12m) 0s ago 29m 60.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (12m) 0s ago 28m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (21m) 0s ago 28m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (11m) 1s ago 26m 84.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (11m) 0s ago 26m 82.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (11m) 1s ago 25m 84.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:15:52.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (11m) 0s ago 25m 84.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:15:53.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:15:53.251 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:15:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:15:53.522 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:15:53.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[100622]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[100622]: from='client.55553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[102844]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:53.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[102844]: from='client.55553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:53.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:53 smithi196 ceph-mon[76349]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:53 smithi196 ceph-mon[76349]: from='client.55553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:15:53.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:53.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:53.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:15:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:15:53.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:15:53.834 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:15:53.834 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:15:53.834 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:15:53.834 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:15:53.834 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:15:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[100622]: from='client.55559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[100622]: from='client.45603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/697286323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2873643337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[102844]: from='client.55559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[102844]: from='client.45603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/697286323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2873643337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:54 smithi196 ceph-mon[76349]: from='client.55559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:54 smithi196 ceph-mon[76349]: from='client.45603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/697286323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2873643337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:15:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:55 smithi196 ceph-mon[76349]: from='client.55577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:55 smithi196 ceph-mon[76349]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:55 smithi184 ceph-mon[102844]: from='client.55577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:55 smithi184 ceph-mon[102844]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:55 smithi184 ceph-mon[100622]: from='client.55577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:55 smithi184 ceph-mon[100622]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:15:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:57 smithi196 ceph-mon[76349]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:57 smithi184 ceph-mon[100622]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:57 smithi184 ceph-mon[102844]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:15:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:59 smithi184 ceph-mon[100622]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:59 smithi184 ceph-mon[102844]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:15:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:59 smithi196 ceph-mon[76349]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:15:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:00.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:00.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:00.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:01 smithi184 ceph-mon[100622]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:01 smithi184 ceph-mon[102844]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:01 smithi196 ceph-mon[76349]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:02] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:16:03.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:03 smithi196 ceph-mon[76349]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:03 smithi184 ceph-mon[100622]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:03.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:03 smithi184 ceph-mon[102844]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:03.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:03.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:03.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:05 smithi184 ceph-mon[100622]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:05 smithi184 ceph-mon[102844]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:05.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:05 smithi196 ceph-mon[76349]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:07 smithi184 ceph-mon[100622]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:07 smithi184 ceph-mon[102844]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:07 smithi196 ceph-mon[76349]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:09 smithi184 ceph-mon[100622]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:09 smithi184 ceph-mon[102844]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:09 smithi196 ceph-mon[76349]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:10.173Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:10.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:10.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:11 smithi184 ceph-mon[100622]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:11 smithi184 ceph-mon[102844]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:11 smithi196 ceph-mon[76349]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:12] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:16:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[100622]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:13.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[102844]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:13 smithi196 ceph-mon[76349]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:13.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:13.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:13.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:15 smithi196 ceph-mon[76349]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:15 smithi184 ceph-mon[102844]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:15 smithi184 ceph-mon[100622]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:17 smithi196 ceph-mon[76349]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:17 smithi184 ceph-mon[102844]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:17 smithi184 ceph-mon[100622]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:19 smithi196 ceph-mon[76349]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:19 smithi184 ceph-mon[102844]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:19 smithi184 ceph-mon[100622]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:20.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:20.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:20.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:21 smithi196 ceph-mon[76349]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:21 smithi184 ceph-mon[100622]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:21 smithi184 ceph-mon[102844]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:22] "GET /metrics HTTP/1.1" 200 38925 "" "Prometheus/2.43.0" 2024-09-20T20:16:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:23 smithi196 ceph-mon[76349]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:23 smithi184 ceph-mon[100622]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:23 smithi184 ceph-mon[102844]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:23.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:23.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:23.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:24.131 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:16:24.659 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:16:24.659 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 33m - - 2024-09-20T20:16:24.659 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 32m - - 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (27m) 13s ago 28m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (28m) 11s ago 28m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (21m) 13s ago 26m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (21m) 11s ago 32m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (22m) 13s ago 33m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (20m) 13s ago 34m 97.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:16:24.660 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (19m) 11s ago 32m 74.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (20m) 13s ago 32m 83.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (27m) 13s ago 27m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (27m) 11s ago 27m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (18m) 13s ago 31m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (17m) 13s ago 31m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:16:24.661 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (17m) 13s ago 31m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (16m) 13s ago 30m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (15m) 11s ago 30m 61.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (14m) 11s ago 30m 57.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (13m) 11s ago 29m 60.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (12m) 11s ago 29m 65.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:16:24.662 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (21m) 11s ago 28m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:16:24.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (12m) 13s ago 26m 84.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:16:24.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (12m) 11s ago 26m 82.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:16:24.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (12m) 13s ago 26m 84.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:16:24.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (12m) 11s ago 26m 84.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:16:24.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:16:24.971 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:16:25.241 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:16:25.242 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:16:25.243 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:16:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[100622]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[100622]: from='client.45627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[100622]: from='client.45633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2135172682' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[102844]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[102844]: from='client.45627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[102844]: from='client.45633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2135172682' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:25.557 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:16:25.557 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:16:25.557 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:16:25.557 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:16:25.557 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:16:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:25 smithi196 ceph-mon[76349]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:25 smithi196 ceph-mon[76349]: from='client.45627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:25 smithi196 ceph-mon[76349]: from='client.45633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:25.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2135172682' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[100622]: from='client.55601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[100622]: from='client.55613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3151813060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[102844]: from='client.55601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[102844]: from='client.55613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3151813060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:26 smithi196 ceph-mon[76349]: from='client.55601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:26 smithi196 ceph-mon[76349]: from='client.55613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3151813060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:27 smithi184 ceph-mon[102844]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:27 smithi184 ceph-mon[100622]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:27 smithi196 ceph-mon[76349]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:29.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:29 smithi184 ceph-mon[100622]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:29.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:29.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:29 smithi184 ceph-mon[102844]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:29.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:29 smithi196 ceph-mon[76349]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:30.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:30.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:30.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:31 smithi196 ceph-mon[76349]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:31 smithi184 ceph-mon[100622]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:31 smithi184 ceph-mon[102844]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:16:32.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[102844]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[100622]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:33.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:33.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:33.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:33 smithi196 ceph-mon[76349]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:35 smithi184 ceph-mon[100622]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:35 smithi184 ceph-mon[102844]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:35 smithi196 ceph-mon[76349]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:37 smithi184 ceph-mon[100622]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:37 smithi184 ceph-mon[102844]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:37 smithi196 ceph-mon[76349]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:39 smithi184 ceph-mon[100622]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:39 smithi184 ceph-mon[102844]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:39 smithi196 ceph-mon[76349]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:40.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:40.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:40.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:41 smithi184 ceph-mon[100622]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:41 smithi184 ceph-mon[102844]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:41 smithi196 ceph-mon[76349]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:16:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:43 smithi184 ceph-mon[100622]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:43 smithi184 ceph-mon[102844]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:43.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:43.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:43.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:43 smithi196 ceph-mon[76349]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:44.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:45 smithi184 ceph-mon[100622]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:45 smithi184 ceph-mon[102844]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:45 smithi196 ceph-mon[76349]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:47 smithi184 ceph-mon[102844]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:47 smithi184 ceph-mon[100622]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:47 smithi196 ceph-mon[76349]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:49.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:49 smithi184 ceph-mon[100622]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:49.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:49 smithi184 ceph-mon[102844]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:49 smithi196 ceph-mon[76349]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:50.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:50.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:50.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:50.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:50.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:50.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:51 smithi184 ceph-mon[102844]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:51 smithi184 ceph-mon[100622]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:51 smithi196 ceph-mon[76349]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:16:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:16:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:52 smithi196 ceph-mon[76349]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[102844]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:52 smithi184 ceph-mon[100622]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:53.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:53.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:16:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:16:53.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:16:55.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:55 smithi196 ceph-mon[76349]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:55 smithi184 ceph-mon[100622]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:55 smithi184 ceph-mon[102844]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:55.857 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:16:56.384 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:16:56.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 33m - - 2024-09-20T20:16:56.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 33m - - 2024-09-20T20:16:56.384 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (27m) 4s ago 28m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:16:56.384 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (28m) 3s ago 28m 90.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (21m) 4s ago 26m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (21m) 3s ago 32m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (22m) 4s ago 34m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (20m) 4s ago 34m 98.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (20m) 3s ago 33m 76.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (20m) 4s ago 33m 84.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:16:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (28m) 4s ago 28m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (28m) 3s ago 28m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (19m) 4s ago 32m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (18m) 4s ago 32m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (17m) 4s ago 31m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (17m) 4s ago 31m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (15m) 3s ago 30m 61.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:16:56.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (15m) 3s ago 30m 57.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (13m) 3s ago 30m 60.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (13m) 3s ago 29m 65.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (22m) 3s ago 29m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (12m) 4s ago 27m 85.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (12m) 3s ago 27m 83.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (12m) 4s ago 27m 85.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:16:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (12m) 3s ago 27m 84.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:16:56.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:16:56.694 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:16:56.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:16:56.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:16:57.271 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:16:57.271 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:16:57.271 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:16:57.271 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:16:57.271 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.45663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.45669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.45675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2158018327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3956567405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.45663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.45669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.45675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2158018327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3956567405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.45663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.45669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.45675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2158018327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:16:57.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3956567405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:16:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:58 smithi196 ceph-mon[76349]: from='client.55649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:58 smithi184 ceph-mon[100622]: from='client.55649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:58 smithi184 ceph-mon[102844]: from='client.55649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:59 smithi196 ceph-mon[76349]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:59 smithi184 ceph-mon[100622]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:16:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:59 smithi184 ceph-mon[102844]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:00.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:00.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:00.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:01 smithi196 ceph-mon[76349]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:01 smithi184 ceph-mon[100622]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:01 smithi184 ceph-mon[102844]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:02] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:17:03.605 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:03 smithi184 ceph-mon[100622]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:03.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:03 smithi184 ceph-mon[102844]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:03 smithi196 ceph-mon[76349]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:03.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:03.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:03.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:05.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:05 smithi196 ceph-mon[76349]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:05 smithi184 ceph-mon[100622]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:05 smithi184 ceph-mon[102844]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:06.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:06 smithi196 ceph-mon[76349]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:06.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:06 smithi184 ceph-mon[100622]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:06 smithi184 ceph-mon[102844]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:08 smithi196 ceph-mon[76349]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:09.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:08 smithi184 ceph-mon[100622]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:09.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:08 smithi184 ceph-mon[102844]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:10.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:10.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:10.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:10 smithi184 ceph-mon[102844]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:10 smithi184 ceph-mon[100622]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:10 smithi196 ceph-mon[76349]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:12] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:17:12.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[100622]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:12.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:12 smithi184 ceph-mon[102844]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:12 smithi196 ceph-mon[76349]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:13.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:13.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:13.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:15 smithi184 ceph-mon[100622]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:15 smithi184 ceph-mon[102844]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:15 smithi196 ceph-mon[76349]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:17 smithi184 ceph-mon[100622]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:17 smithi184 ceph-mon[102844]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:17 smithi196 ceph-mon[76349]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:19 smithi184 ceph-mon[100622]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:19 smithi184 ceph-mon[102844]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:19 smithi196 ceph-mon[76349]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:20.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:20.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:20.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:21 smithi184 ceph-mon[100622]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:21 smithi184 ceph-mon[102844]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:21 smithi196 ceph-mon[76349]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:22] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:17:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:23 smithi184 ceph-mon[100622]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:23 smithi184 ceph-mon[102844]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:23.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:23.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:23.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:23 smithi196 ceph-mon[76349]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:25.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:25 smithi196 ceph-mon[76349]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:25 smithi184 ceph-mon[100622]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:25 smithi184 ceph-mon[102844]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:17:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:27 smithi196 ceph-mon[76349]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:27 smithi184 ceph-mon[100622]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:27 smithi184 ceph-mon[102844]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 34m - - 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 33m - - 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (28m) 16s ago 29m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (29m) 14s ago 29m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (22m) 16s ago 27m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (22m) 14s ago 33m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (23m) 16s ago 35m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (21m) 16s ago 35m 99.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (20m) 14s ago 33m 76.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:17:28.103 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (21m) 16s ago 33m 85.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (29m) 16s ago 29m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (28m) 14s ago 28m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (19m) 16s ago 33m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (18m) 16s ago 32m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (18m) 16s ago 32m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (17m) 16s ago 31m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (16m) 14s ago 31m 61.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (15m) 14s ago 31m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (14m) 14s ago 30m 60.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (13m) 14s ago 30m 65.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (22m) 14s ago 29m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (13m) 16s ago 27m 85.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (13m) 14s ago 28m 83.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (13m) 16s ago 27m 85.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:17:28.104 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (13m) 14s ago 27m 84.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:17:28.414 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:17:28.414 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:17:28.414 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:17:28.415 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:17:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3880572100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:17:28.698 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:17:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:17:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:17:28.699 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:17:28.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3880572100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:17:28.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3880572100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:17:29.017 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:17:29.017 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:17:29.017 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:17:29.017 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:17:29.018 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:17:29.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: from='client.55661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: from='client.55667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: from='client.55673 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1702945170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: from='client.55661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: from='client.55667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: from='client.55673 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:29.671 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1702945170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:17:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: from='client.55661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: from='client.55667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: from='client.55673 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1702945170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:17:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:30.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:30.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:30.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:30.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:30.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:30 smithi184 ceph-mon[100622]: from='client.45723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:30.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:30 smithi184 ceph-mon[102844]: from='client.45723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:30 smithi196 ceph-mon[76349]: from='client.45723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:17:31.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:31 smithi184 ceph-mon[100622]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:31.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:31 smithi184 ceph-mon[102844]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:31 smithi196 ceph-mon[76349]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:32] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:17:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:32 smithi196 ceph-mon[76349]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[100622]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[102844]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:33.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:33.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:33.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:33.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:34.994 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:34.994 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:34.994 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:34 smithi196 ceph-mon[76349]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[100622]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:34 smithi184 ceph-mon[102844]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:36 smithi196 ceph-mon[76349]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:36 smithi184 ceph-mon[100622]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:36 smithi184 ceph-mon[102844]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:38 smithi196 ceph-mon[76349]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:38 smithi184 ceph-mon[100622]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:38 smithi184 ceph-mon[102844]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:40.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:40.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:40.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:40 smithi196 ceph-mon[76349]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:40 smithi184 ceph-mon[100622]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:40 smithi184 ceph-mon[102844]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:42] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:17:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:42 smithi184 ceph-mon[100622]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:42 smithi184 ceph-mon[102844]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:42 smithi196 ceph-mon[76349]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:43.900 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:43.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:43.900 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:43.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:43.900 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:43.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:44 smithi184 ceph-mon[100622]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:44 smithi184 ceph-mon[102844]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:44 smithi196 ceph-mon[76349]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:47.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:46 smithi196 ceph-mon[76349]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:47.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:46 smithi184 ceph-mon[100622]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:46 smithi184 ceph-mon[102844]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:48 smithi184 ceph-mon[100622]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:48 smithi184 ceph-mon[102844]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:48 smithi196 ceph-mon[76349]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:50.507 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:50.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:50.507 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:50.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:50.507 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:50.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:51.116 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:50 smithi184 ceph-mon[100622]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:51.117 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:50 smithi184 ceph-mon[102844]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:50 smithi196 ceph-mon[76349]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:17:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[100622]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[102844]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:52 smithi196 ceph-mon[76349]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:53.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:53.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:17:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:17:53.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:17:55.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:54 smithi196 ceph-mon[76349]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[100622]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:17:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:54 smithi184 ceph-mon[102844]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:56 smithi196 ceph-mon[76349]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:56 smithi184 ceph-mon[100622]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:56 smithi184 ceph-mon[102844]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:59.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:58 smithi196 ceph-mon[76349]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:59.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:58 smithi184 ceph-mon[100622]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:58 smithi184 ceph-mon[102844]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:17:59.314 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:17:59.842 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:17:59.842 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 34m - - 2024-09-20T20:17:59.842 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 34m - - 2024-09-20T20:17:59.842 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (28m) 7s ago 29m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (29m) 6s ago 29m 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (22m) 7s ago 27m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (22m) 6s ago 33m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (23m) 7s ago 35m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (21m) 7s ago 35m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (21m) 6s ago 34m 78.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (21m) 7s ago 34m 86.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (29m) 7s ago 29m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:17:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (29m) 6s ago 29m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (20m) 7s ago 33m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (19m) 7s ago 33m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (18m) 7s ago 32m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (18m) 7s ago 32m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (16m) 6s ago 31m 61.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (16m) 6s ago 31m 57.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (14m) 6s ago 31m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (14m) 6s ago 30m 65.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:17:59.844 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (23m) 6s ago 30m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:17:59.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (13m) 7s ago 28m 85.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:17:59.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (13m) 6s ago 28m 83.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:17:59.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (13m) 7s ago 28m 85.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:17:59.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (13m) 6s ago 28m 85.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:18:00.152 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:18:00.152 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:18:00.153 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:18:00.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:17:59 smithi184 ceph-mon[100622]: from='client.45735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:00.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:17:59 smithi184 ceph-mon[102844]: from='client.45735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:17:59 smithi196 ceph-mon[76349]: from='client.45735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:18:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:18:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:18:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:18:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:18:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:18:00.421 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:18:00.421 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:00.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:00.421 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:00.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:00.421 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:00.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:18:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:18:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:18:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:18:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:18:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: from='client.45741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: from='client.55709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/299359996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: from='client.55721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2182648769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: from='client.45741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: from='client.55709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/299359996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: from='client.55721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2182648769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: from='client.45741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: from='client.55709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/299359996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: from='client.55721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2182648769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:18:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:02 smithi184 ceph-mon[100622]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:02 smithi184 ceph-mon[102844]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:02 smithi196 ceph-mon[76349]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:03.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:03.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:03.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:04 smithi184 ceph-mon[102844]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:04 smithi184 ceph-mon[100622]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:04 smithi196 ceph-mon[76349]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:06.995 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:06 smithi196 ceph-mon[76349]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:06.995 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:06 smithi184 ceph-mon[102844]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:06 smithi184 ceph-mon[100622]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:08 smithi184 ceph-mon[100622]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:08 smithi184 ceph-mon[102844]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:08 smithi196 ceph-mon[76349]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:10.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:10.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:10.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:10.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:10 smithi184 ceph-mon[100622]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:10 smithi184 ceph-mon[102844]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:10 smithi196 ceph-mon[76349]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:12.112 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:18:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:13 smithi196 ceph-mon[76349]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[100622]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[102844]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:13.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:13.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:13.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:14.100 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:18:13.81273607Z level=info msg="Completed cleanup jobs" duration=13.127925ms 2024-09-20T20:18:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:15 smithi196 ceph-mon[76349]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:15 smithi184 ceph-mon[100622]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:15 smithi184 ceph-mon[102844]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:17 smithi196 ceph-mon[76349]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:17 smithi184 ceph-mon[102844]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:17 smithi184 ceph-mon[100622]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:19 smithi196 ceph-mon[76349]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:19 smithi184 ceph-mon[100622]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:19 smithi184 ceph-mon[102844]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:20.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:20.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:20.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:21 smithi196 ceph-mon[76349]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:21 smithi184 ceph-mon[100622]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:21 smithi184 ceph-mon[102844]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:22] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:18:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:23 smithi196 ceph-mon[76349]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:23 smithi184 ceph-mon[100622]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:23 smithi184 ceph-mon[102844]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:23.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:23.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:23.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:25 smithi196 ceph-mon[76349]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:25 smithi184 ceph-mon[100622]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:25 smithi184 ceph-mon[102844]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:26 smithi196 ceph-mon[76349]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:26 smithi184 ceph-mon[100622]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:26 smithi184 ceph-mon[102844]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:28 smithi184 ceph-mon[100622]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:28 smithi184 ceph-mon[102844]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:29.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:28 smithi196 ceph-mon[76349]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:29.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:30.498 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:30.183Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:30.498 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:30.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:30.498 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:30.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:31.036 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:18:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:30 smithi184 ceph-mon[100622]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:30 smithi184 ceph-mon[102844]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:30 smithi196 ceph-mon[76349]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:31.578 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:18:31.578 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 35m - - 2024-09-20T20:18:31.578 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 35m - - 2024-09-20T20:18:31.578 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (29m) 19s ago 30m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (30m) 17s ago 30m 89.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (23m) 19s ago 28m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (23m) 17s ago 34m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (24m) 19s ago 36m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (22m) 19s ago 36m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (21m) 17s ago 34m 78.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (22m) 19s ago 34m 86.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (30m) 19s ago 30m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (30m) 17s ago 30m 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (20m) 19s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (19m) 19s ago 33m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (19m) 19s ago 33m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (18m) 19s ago 32m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:18:31.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (17m) 17s ago 32m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (16m) 17s ago 32m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (15m) 17s ago 31m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (14m) 17s ago 31m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (23m) 17s ago 30m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (14m) 19s ago 29m 85.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (14m) 17s ago 29m 83.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (14m) 19s ago 28m 85.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:18:31.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (14m) 17s ago 28m 85.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:18:31.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:18:31.901 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:18:31.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:18:31.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:18:31.901 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:18:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[100622]: from='client.45771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[100622]: from='client.45777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1070228890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[102844]: from='client.45771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[102844]: from='client.45777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1070228890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:32.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:18:32.231 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:18:32.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:32] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:18:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:31 smithi196 ceph-mon[76349]: from='client.45771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:31 smithi196 ceph-mon[76349]: from='client.45777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1070228890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:18:32.569 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:18:32.569 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:18:32.569 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:18:32.569 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:18:32.569 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='client.55745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='client.45795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2013485211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='client.55745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='client.45795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2013485211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='client.55745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='client.45795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2013485211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:18:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:33.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:33.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:33.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[100622]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[102844]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:34 smithi196 ceph-mon[76349]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:36 smithi196 ceph-mon[76349]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:36 smithi184 ceph-mon[100622]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:36 smithi184 ceph-mon[102844]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:38 smithi184 ceph-mon[100622]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:38 smithi184 ceph-mon[102844]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:38 smithi196 ceph-mon[76349]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:40.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:40.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:40.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:40 smithi184 ceph-mon[100622]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:41.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:40 smithi184 ceph-mon[102844]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:40 smithi196 ceph-mon[76349]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:42] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:18:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:42 smithi184 ceph-mon[100622]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:42 smithi184 ceph-mon[102844]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:42 smithi196 ceph-mon[76349]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:43.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:43.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:43.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:44 smithi184 ceph-mon[100622]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:44 smithi184 ceph-mon[102844]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:44 smithi196 ceph-mon[76349]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:46 smithi184 ceph-mon[100622]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:46 smithi184 ceph-mon[102844]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:46 smithi196 ceph-mon[76349]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:48.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:48 smithi196 ceph-mon[76349]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:48 smithi184 ceph-mon[102844]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:48 smithi184 ceph-mon[100622]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:50.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:50.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:50.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:51.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:50 smithi184 ceph-mon[102844]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:51.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:50 smithi184 ceph-mon[100622]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:50 smithi196 ceph-mon[76349]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:18:52] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[100622]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[102844]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:53 smithi196 ceph-mon[76349]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:53.977 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:53.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:53.977 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:53.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:18:53.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:18:53.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:18:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:55 smithi196 ceph-mon[76349]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[100622]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[102844]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:18:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:57 smithi196 ceph-mon[76349]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:57 smithi184 ceph-mon[100622]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:57 smithi184 ceph-mon[102844]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:59 smithi196 ceph-mon[76349]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:59 smithi184 ceph-mon[100622]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:18:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:59 smithi184 ceph-mon[102844]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:00.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:00.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:00.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:01 smithi196 ceph-mon[76349]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:01 smithi184 ceph-mon[100622]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:01 smithi184 ceph-mon[102844]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:02] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:19:02.865 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:19:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:03 smithi184 ceph-mon[100622]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:03 smithi184 ceph-mon[102844]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:03 smithi196 ceph-mon[76349]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:03.397 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 35m - - 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 35m - - 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (29m) 11s ago 31m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (30m) 9s ago 30m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (23m) 11s ago 29m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (23m) 9s ago 34m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (24m) 11s ago 36m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (23m) 11s ago 36m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (22m) 9s ago 35m 77.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (22m) 11s ago 35m 84.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (30m) 11s ago 30m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (30m) 9s ago 30m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:19:03.398 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (21m) 11s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (20m) 11s ago 34m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (19m) 11s ago 33m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (19m) 11s ago 33m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (17m) 9s ago 33m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (17m) 9s ago 32m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (16m) 9s ago 32m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (15m) 9s ago 31m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (24m) 9s ago 31m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (15m) 11s ago 29m 86.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (14m) 9s ago 29m 84.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (14m) 11s ago 29m 86.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:19:03.399 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (14m) 9s ago 29m 85.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:19:03.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:19:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:19:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:19:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:19:03.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:19:03.715 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:03.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:03.715 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:03.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:03.715 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:03.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:03.982 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:19:03.982 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:19:03.983 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:19:04.293 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:19:04.293 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:19:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:19:04.293 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:19:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[100622]: from='client.55769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[100622]: from='client.55775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[100622]: from='client.55781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2690092368' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[102844]: from='client.55769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[102844]: from='client.55775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[102844]: from='client.55781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2690092368' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:04 smithi196 ceph-mon[76349]: from='client.55769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:04 smithi196 ceph-mon[76349]: from='client.55775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:04 smithi196 ceph-mon[76349]: from='client.55781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2690092368' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:05 smithi196 ceph-mon[76349]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:05 smithi196 ceph-mon[76349]: from='client.55793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3374378113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[100622]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[100622]: from='client.55793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3374378113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[102844]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[102844]: from='client.55793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3374378113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:07 smithi196 ceph-mon[76349]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:07 smithi184 ceph-mon[100622]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:07 smithi184 ceph-mon[102844]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:09 smithi196 ceph-mon[76349]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:09 smithi184 ceph-mon[100622]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:09 smithi184 ceph-mon[102844]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:10.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:10.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:10.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:11.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:11 smithi196 ceph-mon[76349]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:11 smithi184 ceph-mon[100622]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:11 smithi184 ceph-mon[102844]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:12] "GET /metrics HTTP/1.1" 200 38926 "" "Prometheus/2.43.0" 2024-09-20T20:19:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[100622]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[102844]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:13.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:13.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:13.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:13.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:13 smithi196 ceph-mon[76349]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:15 smithi184 ceph-mon[100622]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:15 smithi184 ceph-mon[102844]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:15 smithi196 ceph-mon[76349]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:17 smithi184 ceph-mon[100622]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:17 smithi184 ceph-mon[102844]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:17 smithi196 ceph-mon[76349]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:19 smithi184 ceph-mon[100622]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:19 smithi184 ceph-mon[102844]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:19 smithi196 ceph-mon[76349]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:20.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:20.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:20.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:21 smithi184 ceph-mon[100622]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:21 smithi184 ceph-mon[102844]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:21 smithi196 ceph-mon[76349]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:22] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T20:19:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:23 smithi184 ceph-mon[100622]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:23 smithi184 ceph-mon[102844]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:23.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:23.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:23.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:23 smithi196 ceph-mon[76349]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:25 smithi184 ceph-mon[100622]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:25 smithi184 ceph-mon[102844]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:25 smithi196 ceph-mon[76349]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:27 smithi184 ceph-mon[100622]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:27 smithi184 ceph-mon[102844]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:27.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:27 smithi196 ceph-mon[76349]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:29 smithi184 ceph-mon[100622]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:29 smithi184 ceph-mon[102844]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:29 smithi196 ceph-mon[76349]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:30.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:30.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:30.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:31.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:31 smithi196 ceph-mon[76349]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:31 smithi184 ceph-mon[100622]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:31 smithi184 ceph-mon[102844]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:19:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[100622]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[102844]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.751 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:33.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:33.751 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:33.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:33.751 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:33.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:33 smithi196 ceph-mon[76349]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:34.585 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:19:35.137 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:19:35.137 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 36m - - 2024-09-20T20:19:35.137 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 36m - - 2024-09-20T20:19:35.137 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (30m) 2s ago 31m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (31m) 0s ago 31m 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (24m) 2s ago 29m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (24m) 0s ago 35m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (25m) 2s ago 37m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (23m) 2s ago 37m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (22m) 0s ago 35m 78.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (23m) 2s ago 35m 85.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (31m) 2s ago 31m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (31m) 0s ago 31m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (21m) 2s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (20m) 2s ago 34m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:19:35.138 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (20m) 2s ago 34m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (19m) 2s ago 33m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (18m) 0s ago 33m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (17m) 0s ago 33m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (16m) 0s ago 32m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (15m) 0s ago 32m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (24m) 0s ago 31m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (15m) 2s ago 30m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (15m) 0s ago 30m 84.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (15m) 2s ago 29m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:19:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (15m) 0s ago 29m 86.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:19:35.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:19:35.449 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:19:35.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[100622]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[102844]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:35 smithi196 ceph-mon[76349]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:35.716 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:19:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:19:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:19:35.717 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:19:36.028 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:19:36.028 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:19:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:19:36.028 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:19:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:19:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:36 smithi196 ceph-mon[76349]: from='client.55805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:36 smithi196 ceph-mon[76349]: from='client.55811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:36 smithi196 ceph-mon[76349]: from='client.45855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3244872662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3497631302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[102844]: from='client.55805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[102844]: from='client.55811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[102844]: from='client.45855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3244872662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3497631302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[100622]: from='client.55805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[100622]: from='client.55811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[100622]: from='client.45855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3244872662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:19:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3497631302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:19:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:37 smithi196 ceph-mon[76349]: from='client.45867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:37.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:37 smithi196 ceph-mon[76349]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[100622]: from='client.45867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[100622]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[102844]: from='client.45867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[102844]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:39 smithi196 ceph-mon[76349]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:39 smithi184 ceph-mon[100622]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:39 smithi184 ceph-mon[102844]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:40.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:40.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:40.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:41 smithi196 ceph-mon[76349]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:41 smithi184 ceph-mon[100622]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:41 smithi184 ceph-mon[102844]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T20:19:43.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:43 smithi184 ceph-mon[100622]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:43.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:43 smithi184 ceph-mon[102844]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:43 smithi196 ceph-mon[76349]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:43.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:43.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:43.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:45 smithi196 ceph-mon[76349]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:45 smithi184 ceph-mon[100622]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:45 smithi184 ceph-mon[102844]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:47 smithi196 ceph-mon[76349]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:47 smithi184 ceph-mon[100622]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:47 smithi184 ceph-mon[102844]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:49 smithi196 ceph-mon[76349]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:49 smithi184 ceph-mon[100622]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:49 smithi184 ceph-mon[102844]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:50.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:50.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:50.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:51 smithi196 ceph-mon[76349]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:51 smithi184 ceph-mon[100622]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:51 smithi184 ceph-mon[102844]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:19:52] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:19:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[102844]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[100622]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:53 smithi196 ceph-mon[76349]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:53.994 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:53.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:53.994 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:53.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:19:53.994 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:19:53.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:19:55.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:55 smithi196 ceph-mon[76349]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:55.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[100622]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[102844]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:19:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:57 smithi184 ceph-mon[100622]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:57 smithi184 ceph-mon[102844]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:57.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:57 smithi196 ceph-mon[76349]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:59 smithi184 ceph-mon[100622]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:59 smithi184 ceph-mon[102844]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:19:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:59 smithi196 ceph-mon[76349]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:00.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:00.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:00.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:01 smithi184 ceph-mon[100622]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:01 smithi184 ceph-mon[102844]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:01 smithi196 ceph-mon[76349]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:02] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:20:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:03 smithi184 ceph-mon[100622]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:03 smithi184 ceph-mon[102844]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:03.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:03.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:03.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:03 smithi196 ceph-mon[76349]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:05 smithi184 ceph-mon[100622]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:05 smithi184 ceph-mon[102844]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:05 smithi196 ceph-mon[76349]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:06.324 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:20:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:20:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 36m - - 2024-09-20T20:20:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 36m - - 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (31m) 14s ago 32m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (31m) 12s ago 31m 89.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (25m) 14s ago 30m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (24m) 12s ago 35m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (25m) 14s ago 37m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (24m) 14s ago 37m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (23m) 12s ago 36m 78.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (23m) 14s ago 36m 86.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (31m) 14s ago 31m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (31m) 12s ago 31m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (22m) 14s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (21m) 14s ago 35m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (20m) 14s ago 34m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (20m) 14s ago 34m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:20:06.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (19m) 12s ago 34m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (18m) 12s ago 33m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (17m) 12s ago 33m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (16m) 12s ago 32m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (25m) 12s ago 32m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (16m) 14s ago 30m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (15m) 12s ago 30m 84.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (15m) 14s ago 30m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:20:06.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (15m) 12s ago 30m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:07.165 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:20:07.166 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:20:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:20:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:20:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:20:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:20:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:20:07.436 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:20:07.748 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:07.748 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:07.748 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:20:07.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:20:07.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[100622]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:07.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[100622]: from='client.45876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3637146494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[102844]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[102844]: from='client.45876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:07.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3637146494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:07 smithi196 ceph-mon[76349]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:07 smithi196 ceph-mon[76349]: from='client.45876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3637146494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[100622]: from='client.55847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[100622]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[100622]: from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2368771012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[102844]: from='client.55847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[102844]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[102844]: from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2368771012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:08 smithi196 ceph-mon[76349]: from='client.55847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:08 smithi196 ceph-mon[76349]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:08 smithi196 ceph-mon[76349]: from='client.45900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:08.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2368771012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:09 smithi184 ceph-mon[100622]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:09 smithi184 ceph-mon[102844]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:09 smithi196 ceph-mon[76349]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:10.190Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:10.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:10.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:11.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:11 smithi196 ceph-mon[76349]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:11 smithi184 ceph-mon[100622]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:11 smithi184 ceph-mon[102844]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:12] "GET /metrics HTTP/1.1" 200 38928 "" "Prometheus/2.43.0" 2024-09-20T20:20:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:13 smithi196 ceph-mon[76349]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[100622]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[102844]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:13.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:13.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:13.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:13.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:13.774 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:13.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:15.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[100622]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:15.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[102844]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:15.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:15 smithi196 ceph-mon[76349]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:17 smithi184 ceph-mon[100622]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:17 smithi184 ceph-mon[102844]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:17 smithi196 ceph-mon[76349]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:19 smithi184 ceph-mon[100622]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:19 smithi184 ceph-mon[102844]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:19 smithi196 ceph-mon[76349]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:20.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:20.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:20.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:21 smithi184 ceph-mon[100622]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:21 smithi184 ceph-mon[102844]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:21 smithi196 ceph-mon[76349]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:22] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T20:20:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:23 smithi184 ceph-mon[100622]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:23 smithi184 ceph-mon[102844]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:23.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:23.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:23.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:23 smithi196 ceph-mon[76349]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:25 smithi184 ceph-mon[100622]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:25 smithi184 ceph-mon[102844]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:25 smithi196 ceph-mon[76349]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:27 smithi184 ceph-mon[100622]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:27 smithi184 ceph-mon[102844]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:27 smithi196 ceph-mon[76349]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:29 smithi184 ceph-mon[100622]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:29 smithi184 ceph-mon[102844]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:29 smithi196 ceph-mon[76349]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:30.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:30.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:30.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:31 smithi184 ceph-mon[100622]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:31 smithi184 ceph-mon[102844]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:31.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:31 smithi196 ceph-mon[76349]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:20:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:33 smithi196 ceph-mon[76349]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[102844]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[100622]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:33.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:33.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:33.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[100622]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[102844]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:35 smithi196 ceph-mon[76349]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:36 smithi184 ceph-mon[102844]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:36 smithi184 ceph-mon[100622]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:36 smithi196 ceph-mon[76349]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:38.043 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:20:38.569 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 37m - - 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 37m - - 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (31m) 5s ago 32m 30.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (32m) 3s ago 32m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (25m) 5s ago 30m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (25m) 3s ago 36m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (26m) 5s ago 38m 536M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (24m) 5s ago 38m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (24m) 3s ago 36m 80.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (24m) 5s ago 36m 88.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (32m) 5s ago 32m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (32m) 3s ago 32m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (22m) 5s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:20:38.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (22m) 5s ago 35m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (21m) 5s ago 35m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (20m) 5s ago 35m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (19m) 3s ago 34m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (18m) 3s ago 34m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (17m) 3s ago 33m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (16m) 3s ago 33m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (25m) 3s ago 33m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (16m) 5s ago 31m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (16m) 3s ago 31m 85.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (16m) 5s ago 30m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:20:38.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (16m) 3s ago 30m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:20:38.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:20:38.878 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:20:39.148 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:20:39.148 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:20:39.148 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:20:39.148 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:20:39.149 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:20:39.149 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[100622]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[100622]: from='client.45912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[100622]: from='client.55883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2599939135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[102844]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[102844]: from='client.45912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[102844]: from='client.55883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.150 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2599939135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:38 smithi196 ceph-mon[76349]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:38 smithi196 ceph-mon[76349]: from='client.45912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:38 smithi196 ceph-mon[76349]: from='client.55883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2599939135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:20:39.459 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:20:39.459 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:39.459 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:39.459 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:20:39.459 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[100622]: from='client.55889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[100622]: from='client.45936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3363256778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[102844]: from='client.55889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[102844]: from='client.45936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3363256778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:40.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:40.192Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:40.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:40.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:40.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:40.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:39 smithi196 ceph-mon[76349]: from='client.55889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:39 smithi196 ceph-mon[76349]: from='client.45936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3363256778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:20:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:40 smithi184 ceph-mon[100622]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:40 smithi184 ceph-mon[102844]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:40 smithi196 ceph-mon[76349]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:20:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:42 smithi184 ceph-mon[100622]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:42 smithi184 ceph-mon[102844]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:42 smithi196 ceph-mon[76349]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:43.955 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:43.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:43.956 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:43.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:43.956 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:43.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:44 smithi184 ceph-mon[100622]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:44 smithi184 ceph-mon[102844]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:44 smithi196 ceph-mon[76349]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:46 smithi184 ceph-mon[100622]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:46 smithi184 ceph-mon[102844]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:46 smithi196 ceph-mon[76349]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:48 smithi184 ceph-mon[100622]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:48 smithi184 ceph-mon[102844]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:48 smithi196 ceph-mon[76349]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:50.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:50.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:50.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:50 smithi184 ceph-mon[100622]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:50 smithi184 ceph-mon[102844]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:50 smithi196 ceph-mon[76349]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:20:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:20:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:52 smithi196 ceph-mon[76349]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:52 smithi184 ceph-mon[100622]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:52 smithi184 ceph-mon[102844]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:53.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:53.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:20:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:20:53.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:20:54.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[100622]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[102844]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:55 smithi196 ceph-mon[76349]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:20:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:20:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:57 smithi196 ceph-mon[76349]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:57 smithi184 ceph-mon[100622]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:57 smithi184 ceph-mon[102844]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:59 smithi184 ceph-mon[100622]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:59 smithi184 ceph-mon[102844]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:20:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:59 smithi196 ceph-mon[76349]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:20:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:00.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:00.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:00.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:01.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:01 smithi196 ceph-mon[76349]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:01 smithi184 ceph-mon[100622]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:01 smithi184 ceph-mon[102844]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:21:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:03 smithi184 ceph-mon[100622]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:03 smithi184 ceph-mon[102844]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:03 smithi196 ceph-mon[76349]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:03.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:03.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:03.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:05 smithi184 ceph-mon[100622]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:05 smithi184 ceph-mon[102844]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:05 smithi196 ceph-mon[76349]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:07 smithi184 ceph-mon[100622]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:07 smithi184 ceph-mon[102844]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:07 smithi196 ceph-mon[76349]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:09 smithi184 ceph-mon[100622]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:09 smithi184 ceph-mon[102844]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:09 smithi196 ceph-mon[76349]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:09.754 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:21:10.280 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 37m - - 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 37m - - 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (32m) 17s ago 33m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (32m) 15s ago 32m 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (26m) 17s ago 31m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (25m) 15s ago 37m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:21:10.281 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (26m) 17s ago 38m 536M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (25m) 17s ago 38m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (24m) 15s ago 37m 76.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (24m) 17s ago 37m 90.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (32m) 17s ago 32m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (32m) 15s ago 32m 16.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (23m) 17s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:21:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (22m) 17s ago 36m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (21m) 17s ago 35m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (21m) 17s ago 35m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (20m) 15s ago 35m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (19m) 15s ago 34m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (18m) 15s ago 34m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (17m) 15s ago 34m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (26m) 15s ago 33m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:21:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (17m) 17s ago 31m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:21:10.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (16m) 15s ago 31m 85.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:21:10.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (16m) 17s ago 31m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:21:10.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (16m) 15s ago 31m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:21:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:10.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:10.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:10.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:10.592 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:10.593 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:21:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:21:10.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:21:10.594 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:21:10.862 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:21:10.863 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:21:10.864 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:21:11.175 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:21:11.175 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:11.175 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:11.175 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:21:11.175 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:21:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[102844]: from='client.45948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[102844]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[102844]: from='client.45954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[102844]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3830015833' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[100622]: from='client.45948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[100622]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[100622]: from='client.45954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[100622]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3830015833' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:11 smithi196 ceph-mon[76349]: from='client.45948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:11 smithi196 ceph-mon[76349]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:11 smithi196 ceph-mon[76349]: from='client.45954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:11 smithi196 ceph-mon[76349]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3830015833' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:12 smithi184 ceph-mon[100622]: from='client.55937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3411672331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:21:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:12 smithi184 ceph-mon[102844]: from='client.55937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3411672331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:12.450 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:12 smithi196 ceph-mon[76349]: from='client.55937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:12.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3411672331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:13.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:13 smithi196 ceph-mon[76349]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:13 smithi184 ceph-mon[100622]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:13 smithi184 ceph-mon[102844]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:13.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:13.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:13.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:14.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[100622]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[102844]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:15 smithi196 ceph-mon[76349]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:17 smithi184 ceph-mon[100622]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:17 smithi184 ceph-mon[102844]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:17 smithi196 ceph-mon[76349]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:19 smithi184 ceph-mon[100622]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:19.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:19 smithi184 ceph-mon[102844]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:19 smithi196 ceph-mon[76349]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:20.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:20.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:20.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:21 smithi184 ceph-mon[100622]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:21 smithi184 ceph-mon[102844]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:21 smithi196 ceph-mon[76349]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:21:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:23 smithi184 ceph-mon[100622]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:23 smithi184 ceph-mon[102844]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:23 smithi196 ceph-mon[76349]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:23.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:23.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:23.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:25 smithi184 ceph-mon[100622]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:25 smithi184 ceph-mon[102844]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:25 smithi196 ceph-mon[76349]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:27 smithi184 ceph-mon[100622]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:27 smithi184 ceph-mon[102844]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:27 smithi196 ceph-mon[76349]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:29 smithi184 ceph-mon[100622]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:29 smithi184 ceph-mon[102844]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:29 smithi196 ceph-mon[76349]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:30.195Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:30.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:30.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:31 smithi184 ceph-mon[100622]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:31 smithi184 ceph-mon[102844]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:31 smithi196 ceph-mon[76349]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:21:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:33 smithi196 ceph-mon[76349]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:33 smithi184 ceph-mon[100622]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:33 smithi184 ceph-mon[102844]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:33.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:33.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:33.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:34.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:35 smithi196 ceph-mon[76349]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[100622]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[102844]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:37.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:37 smithi184 ceph-mon[100622]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:37.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:37 smithi184 ceph-mon[102844]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:37.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:37 smithi196 ceph-mon[76349]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:39 smithi184 ceph-mon[100622]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:39 smithi184 ceph-mon[102844]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:39 smithi196 ceph-mon[76349]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:40.195Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:40.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:40.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:41.476 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:21:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:41 smithi184 ceph-mon[100622]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:41 smithi184 ceph-mon[102844]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:41 smithi196 ceph-mon[76349]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 38m - - 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 38m - - 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (32m) 8s ago 33m 30.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (33m) 6s ago 33m 90.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (26m) 8s ago 31m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:21:42.003 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (26m) 6s ago 37m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (27m) 8s ago 39m 536M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (25m) 8s ago 39m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (25m) 6s ago 37m 78.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (25m) 8s ago 37m 85.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (33m) 8s ago 33m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (33m) 6s ago 33m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (23m) 8s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (23m) 8s ago 36m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (22m) 8s ago 36m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (21m) 8s ago 36m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (20m) 6s ago 35m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (19m) 6s ago 35m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (18m) 6s ago 34m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (17m) 6s ago 34m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:21:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (26m) 6s ago 34m 63.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:21:42.005 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (17m) 8s ago 32m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:21:42.005 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (17m) 6s ago 32m 85.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:21:42.005 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (17m) 8s ago 31m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:21:42.005 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (17m) 6s ago 31m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:21:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:21:42.311 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:42.312 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:21:42.313 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:21:42.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:21:42.588 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:21:42.589 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:21:42.589 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:21:42.589 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:21:42.589 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:21:42.589 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:42 smithi184 ceph-mon[100622]: from='client.55949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:42.590 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/35371347' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:42.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:42 smithi184 ceph-mon[102844]: from='client.55949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:42.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/35371347' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:42 smithi196 ceph-mon[76349]: from='client.55949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/35371347' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:21:42.897 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:21:42.897 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:42.897 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:42.897 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:21:42.897 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[100622]: from='client.55955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[100622]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[100622]: from='client.55961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1195415529' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[102844]: from='client.55955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[102844]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[102844]: from='client.55961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1195415529' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:43.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:43.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:43.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:43.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:43.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:43.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:43 smithi196 ceph-mon[76349]: from='client.55955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:43 smithi196 ceph-mon[76349]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:43 smithi196 ceph-mon[76349]: from='client.55961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1195415529' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:21:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:44 smithi184 ceph-mon[100622]: from='client.46008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:44 smithi184 ceph-mon[102844]: from='client.46008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:44 smithi196 ceph-mon[76349]: from='client.46008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:45.681 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:45 smithi196 ceph-mon[76349]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:45 smithi184 ceph-mon[100622]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:45 smithi184 ceph-mon[102844]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:47 smithi184 ceph-mon[100622]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:47 smithi184 ceph-mon[102844]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:47 smithi196 ceph-mon[76349]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:49 smithi184 ceph-mon[100622]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:49 smithi184 ceph-mon[102844]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:49 smithi196 ceph-mon[76349]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:50.196Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:50.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:50.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:51 smithi184 ceph-mon[100622]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:51 smithi184 ceph-mon[102844]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:51 smithi196 ceph-mon[76349]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:21:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:21:53.719 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:53 smithi196 ceph-mon[76349]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:53 smithi184 ceph-mon[100622]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:53 smithi184 ceph-mon[102844]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:53.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:53.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:21:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:21:53.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:21:54.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[100622]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[102844]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:55 smithi196 ceph-mon[76349]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:55.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:55.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:21:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:57 smithi184 ceph-mon[100622]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:57 smithi184 ceph-mon[102844]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:57 smithi196 ceph-mon[76349]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:59 smithi184 ceph-mon[100622]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:59 smithi184 ceph-mon[102844]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:59 smithi196 ceph-mon[76349]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:00.197Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:00.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:00.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:01 smithi184 ceph-mon[100622]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:01 smithi184 ceph-mon[102844]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:01 smithi196 ceph-mon[76349]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:22:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:03 smithi184 ceph-mon[100622]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:03 smithi184 ceph-mon[102844]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:03.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:03.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:03.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:03 smithi196 ceph-mon[76349]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:05 smithi184 ceph-mon[100622]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:05 smithi184 ceph-mon[102844]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:05 smithi196 ceph-mon[76349]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:07 smithi184 ceph-mon[100622]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:07 smithi184 ceph-mon[102844]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:07 smithi196 ceph-mon[76349]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:09 smithi196 ceph-mon[76349]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:09 smithi184 ceph-mon[100622]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:09 smithi184 ceph-mon[102844]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:10.197Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:10.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:10.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:11 smithi196 ceph-mon[76349]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:11 smithi184 ceph-mon[100622]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:11 smithi184 ceph-mon[102844]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:22:13.195 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 38m - - 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 38m - - 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (33m) 0s ago 34m 30.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (34m) 18s ago 34m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (27m) 0s ago 32m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:22:13.740 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (26m) 18s ago 38m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (27m) 0s ago 39m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (26m) 0s ago 39m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (25m) 18s ago 38m 79.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (25m) 0s ago 38m 88.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (33m) 0s ago 33m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (33m) 18s ago 33m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (24m) 0s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (23m) 0s ago 37m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (22m) 0s ago 36m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (22m) 0s ago 36m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (21m) 18s ago 36m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (20m) 18s ago 35m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (19m) 18s ago 35m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (18m) 18s ago 35m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:22:13.741 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (27m) 18s ago 34m 63.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:22:13.742 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (18m) 0s ago 32m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:22:13.742 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (17m) 18s ago 32m 86.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:22:13.742 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (18m) 0s ago 32m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:22:13.742 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (17m) 18s ago 32m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:22:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:13 smithi196 ceph-mon[76349]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:13 smithi184 ceph-mon[100622]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:13 smithi184 ceph-mon[102844]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:13.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:13.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:13.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:22:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:22:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:22:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:22:14.052 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:22:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:22:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:22:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:22:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:22:14.321 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:22:14.633 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:22:14.633 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:14.633 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:22:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:22:14.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='client.55985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='client.55991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2572869137' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='client.55985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='client.55991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:14.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2572869137' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='client.55985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='client.55991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2572869137' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='client.55997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='client.46044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2455365250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='client.55997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='client.46044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2455365250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='client.55997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='client.46044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:16.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2455365250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:16.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:16.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:16 smithi196 ceph-mon[76349]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:16 smithi184 ceph-mon[100622]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:16 smithi184 ceph-mon[102844]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:17.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:17.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:18.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:18 smithi184 ceph-mon[100622]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:18.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:18 smithi184 ceph-mon[102844]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:18 smithi196 ceph-mon[76349]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:20.198Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:20.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:20.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:20 smithi184 ceph-mon[100622]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:20 smithi184 ceph-mon[102844]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:20 smithi196 ceph-mon[76349]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:22:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:22 smithi184 ceph-mon[100622]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:22 smithi184 ceph-mon[102844]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:22 smithi196 ceph-mon[76349]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:23.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:23.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:23.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:24 smithi184 ceph-mon[100622]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:24 smithi184 ceph-mon[102844]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:24 smithi196 ceph-mon[76349]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:26 smithi196 ceph-mon[76349]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:26 smithi184 ceph-mon[100622]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:26 smithi184 ceph-mon[102844]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:28 smithi196 ceph-mon[76349]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:28 smithi184 ceph-mon[100622]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:28 smithi184 ceph-mon[102844]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:30.199Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:30.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:30.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:30 smithi184 ceph-mon[100622]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:30 smithi184 ceph-mon[102844]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:30 smithi196 ceph-mon[76349]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:22:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:32 smithi184 ceph-mon[100622]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:32 smithi184 ceph-mon[102844]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:32 smithi196 ceph-mon[76349]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:33.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:33.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:33.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:35.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:34 smithi196 ceph-mon[76349]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[100622]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:35.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:34 smithi184 ceph-mon[102844]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.063 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.063 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.063 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.064 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:37 smithi184 ceph-mon[100622]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:37 smithi184 ceph-mon[102844]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:37 smithi196 ceph-mon[76349]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:38.706 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:38.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:39 smithi184 ceph-mon[100622]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:39 smithi184 ceph-mon[102844]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:39 smithi196 ceph-mon[76349]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:40.504 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:40.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:40.504 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:40.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:40.505 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:40.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:41 smithi184 ceph-mon[100622]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:41 smithi184 ceph-mon[102844]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:41 smithi196 ceph-mon[76349]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:22:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:43 smithi184 ceph-mon[100622]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:43 smithi184 ceph-mon[102844]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:43.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:43.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:43.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:43 smithi196 ceph-mon[76349]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:44.930 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 39m - - 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 39m - - 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (33m) 11s ago 34m 30.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (34m) 10s ago 34m 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (27m) 11s ago 32m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (27m) 10s ago 38m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (28m) 11s ago 40m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (26m) 11s ago 40m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (26m) 10s ago 39m 81.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (26m) 11s ago 38m 88.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:22:45.465 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (34m) 11s ago 34m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (34m) 10s ago 34m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (25m) 11s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (24m) 11s ago 37m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (23m) 11s ago 37m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (22m) 11s ago 37m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (21m) 10s ago 36m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (20m) 10s ago 36m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (19m) 10s ago 35m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (19m) 10s ago 35m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (27m) 10s ago 35m 66.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (18m) 11s ago 33m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (18m) 10s ago 33m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (18m) 11s ago 32m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:22:45.466 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (18m) 10s ago 32m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:45.776 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:22:45.777 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:22:45.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:45 smithi184 ceph-mon[100622]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:45.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:45 smithi184 ceph-mon[102844]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:45 smithi196 ceph-mon[76349]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:22:46.044 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:22:46.045 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:22:46.353 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:22:46.353 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:46.353 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:22:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:22:46.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[100622]: from='client.46056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[100622]: from='client.56027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[100622]: from='client.56033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3589113440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1177324903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[102844]: from='client.46056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[102844]: from='client.56027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[102844]: from='client.56033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3589113440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:46.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1177324903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:46 smithi196 ceph-mon[76349]: from='client.46056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:46 smithi196 ceph-mon[76349]: from='client.56027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:46 smithi196 ceph-mon[76349]: from='client.56033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3589113440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1177324903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:22:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:47 smithi184 ceph-mon[100622]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:47 smithi184 ceph-mon[100622]: from='client.46080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:47 smithi184 ceph-mon[102844]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:47 smithi184 ceph-mon[102844]: from='client.46080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:47 smithi196 ceph-mon[76349]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:47 smithi196 ceph-mon[76349]: from='client.46080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:49 smithi184 ceph-mon[100622]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:49 smithi184 ceph-mon[102844]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:49 smithi196 ceph-mon[76349]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:50.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:50.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:50.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:51 smithi184 ceph-mon[100622]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:51 smithi184 ceph-mon[102844]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:51 smithi196 ceph-mon[76349]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:22:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:22:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:53 smithi184 ceph-mon[100622]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:53 smithi184 ceph-mon[102844]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:53.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:53.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:22:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:22:53.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:22:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:53 smithi196 ceph-mon[76349]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[100622]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:54 smithi184 ceph-mon[102844]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:54 smithi196 ceph-mon[76349]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:56.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:56.730 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:56.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:56.731 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:56 smithi196 ceph-mon[76349]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[102844]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:22:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:56 smithi184 ceph-mon[100622]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:57.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:58.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:58 smithi184 ceph-mon[100622]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:58.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:58 smithi184 ceph-mon[102844]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:58 smithi196 ceph-mon[76349]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:59.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:59.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:22:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:00.537 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:00.201Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:00.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:00.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:00.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:00.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:00 smithi184 ceph-mon[100622]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:00 smithi184 ceph-mon[102844]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:00 smithi196 ceph-mon[76349]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:02.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:23:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:02 smithi196 ceph-mon[76349]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:02 smithi184 ceph-mon[100622]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:02 smithi184 ceph-mon[102844]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:03.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:03.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:03.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:04 smithi184 ceph-mon[100622]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:04 smithi184 ceph-mon[102844]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:04 smithi196 ceph-mon[76349]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:06 smithi184 ceph-mon[100622]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:06 smithi184 ceph-mon[102844]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:06 smithi196 ceph-mon[76349]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:09 smithi184 ceph-mon[100622]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:09 smithi184 ceph-mon[102844]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:09 smithi196 ceph-mon[76349]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:10.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:10.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:10.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:11 smithi184 ceph-mon[100622]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:11 smithi184 ceph-mon[102844]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:11 smithi196 ceph-mon[76349]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:23:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:13 smithi184 ceph-mon[100622]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:13 smithi184 ceph-mon[102844]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:13 smithi196 ceph-mon[76349]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:14.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:13.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:14.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:13.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:14.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:13.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[100622]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[102844]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:15 smithi196 ceph-mon[76349]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:16.650 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:23:17.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:16 smithi196 ceph-mon[76349]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[100622]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:16 smithi184 ceph-mon[102844]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:17.177 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 40m - - 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 39m - - 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (34m) 3s ago 35m 30.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (35m) 1s ago 35m 90.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (28m) 3s ago 33m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (28m) 1s ago 39m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (28m) 3s ago 40m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (27m) 3s ago 40m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (26m) 1s ago 39m 82.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (26m) 3s ago 39m 90.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (34m) 3s ago 34m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:23:17.178 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (34m) 1s ago 34m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (25m) 3s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (24m) 3s ago 38m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (24m) 3s ago 38m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (23m) 3s ago 37m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (22m) 1s ago 37m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (21m) 1s ago 36m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (20m) 1s ago 36m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (19m) 1s ago 36m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (28m) 1s ago 35m 63.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (19m) 3s ago 33m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (19m) 1s ago 33m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (19m) 3s ago 33m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:23:17.179 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (18m) 1s ago 33m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:23:17.486 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:23:17.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:23:17.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:23:17.488 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:23:17.758 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:23:17.759 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:23:17.759 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:23:17.759 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:23:17.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:17.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[100622]: from='client.46092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[100622]: from='client.46098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[100622]: from='client.46104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2417796234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:17.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:17.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[102844]: from='client.46092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[102844]: from='client.46098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[102844]: from='client.46104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:17.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2417796234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:18.066 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:23:18.066 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:18.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:18.067 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:23:18.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:23:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:17 smithi196 ceph-mon[76349]: from='client.46092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:17 smithi196 ceph-mon[76349]: from='client.46098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:17 smithi196 ceph-mon[76349]: from='client.46104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2417796234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:18 smithi196 ceph-mon[76349]: from='client.56081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:18 smithi196 ceph-mon[76349]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/103945784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[100622]: from='client.56081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[100622]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/103945784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:19.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[102844]: from='client.56081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:19.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[102844]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:19.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/103945784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:20.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:20.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:20.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:20.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:20.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:20.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:20 smithi184 ceph-mon[100622]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:20 smithi184 ceph-mon[102844]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:20 smithi196 ceph-mon[76349]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:22.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:23:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:22 smithi184 ceph-mon[100622]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:22 smithi184 ceph-mon[102844]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:22 smithi196 ceph-mon[76349]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:23.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:23.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:23.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:25 smithi184 ceph-mon[100622]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:25 smithi184 ceph-mon[102844]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:25 smithi196 ceph-mon[76349]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:27 smithi184 ceph-mon[100622]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:27 smithi184 ceph-mon[102844]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:27 smithi196 ceph-mon[76349]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:29 smithi184 ceph-mon[102844]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:29 smithi184 ceph-mon[100622]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:29 smithi196 ceph-mon[76349]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:30.203Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:30.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:30.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:31 smithi184 ceph-mon[100622]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:31 smithi184 ceph-mon[102844]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:31 smithi196 ceph-mon[76349]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:23:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:33 smithi184 ceph-mon[100622]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:33 smithi184 ceph-mon[102844]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:33 smithi196 ceph-mon[76349]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:33.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:33.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:33.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[102844]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[100622]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:35 smithi196 ceph-mon[76349]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:36.887 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:36.887 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:36.887 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:36.887 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:36 smithi196 ceph-mon[76349]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[100622]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:36 smithi184 ceph-mon[102844]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:38 smithi196 ceph-mon[76349]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:38 smithi184 ceph-mon[100622]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:38 smithi184 ceph-mon[102844]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:40.204Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:40.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:40.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:41.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:41 smithi184 ceph-mon[100622]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:41.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:41 smithi184 ceph-mon[102844]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:41 smithi196 ceph-mon[76349]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:42.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:43 smithi184 ceph-mon[100622]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:43 smithi184 ceph-mon[102844]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:43.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:43 smithi196 ceph-mon[76349]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:44.011 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:43.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:44.011 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:43.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:44.011 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:43.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:45 smithi184 ceph-mon[100622]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:45 smithi184 ceph-mon[102844]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:45 smithi196 ceph-mon[76349]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:47 smithi184 ceph-mon[100622]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:47 smithi184 ceph-mon[102844]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:47 smithi196 ceph-mon[76349]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:48.360 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:23:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:48.885 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 40m - - 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 40m - - 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (34m) 14s ago 35m 30.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (35m) 13s ago 35m 90.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (28m) 14s ago 33m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (28m) 13s ago 39m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (29m) 14s ago 41m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:23:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (27m) 14s ago 41m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (27m) 13s ago 40m 83.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (27m) 14s ago 39m 90.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (35m) 14s ago 35m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (35m) 13s ago 35m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (26m) 14s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (25m) 14s ago 38m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (24m) 14s ago 38m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (24m) 14s ago 38m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (22m) 13s ago 37m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (21m) 13s ago 37m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (20m) 13s ago 37m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:23:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (20m) 13s ago 36m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:23:48.888 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (28m) 13s ago 36m 63.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:23:48.888 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (19m) 14s ago 34m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:23:48.888 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (19m) 13s ago 34m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:23:48.888 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (19m) 14s ago 33m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:23:48.888 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (19m) 13s ago 33m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:23:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:49 smithi196 ceph-mon[76349]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:49 smithi196 ceph-mon[76349]: from='client.56093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:23:49.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:23:49.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:23:49.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:23:49.198 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:23:49.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:49 smithi184 ceph-mon[100622]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:49.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:49 smithi184 ceph-mon[100622]: from='client.56093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:49.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:49 smithi184 ceph-mon[102844]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:49.199 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:49 smithi184 ceph-mon[102844]: from='client.56093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:23:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:23:49.468 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:23:49.777 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:23:49.777 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:49.778 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:23:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[100622]: from='client.56099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[100622]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1040187027' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[100622]: from='client.56117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2935679739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[102844]: from='client.56099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[102844]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1040187027' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[102844]: from='client.56117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2935679739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:50.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:50.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:50.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:50.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:50.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:50.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:50 smithi196 ceph-mon[76349]: from='client.56099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:50 smithi196 ceph-mon[76349]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1040187027' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:50 smithi196 ceph-mon[76349]: from='client.56117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2935679739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:51 smithi184 ceph-mon[100622]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:51 smithi184 ceph-mon[102844]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:51 smithi196 ceph-mon[76349]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:23:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:23:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:53 smithi184 ceph-mon[100622]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:53 smithi184 ceph-mon[102844]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:53 smithi196 ceph-mon[76349]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:53.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:53.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:23:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:23:53.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[100622]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[102844]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:55 smithi196 ceph-mon[76349]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:56 smithi196 ceph-mon[76349]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:57.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[100622]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[102844]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:57.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:23:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:58.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:59.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:58 smithi196 ceph-mon[76349]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:59.202 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:58 smithi184 ceph-mon[100622]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:58 smithi184 ceph-mon[102844]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:00.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:00.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:00.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:00 smithi184 ceph-mon[100622]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:00 smithi184 ceph-mon[102844]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:00 smithi196 ceph-mon[76349]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:24:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:02 smithi184 ceph-mon[100622]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:02 smithi184 ceph-mon[102844]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:02 smithi196 ceph-mon[76349]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:03.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:03.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:03.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:04 smithi184 ceph-mon[100622]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:24:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:04 smithi184 ceph-mon[102844]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:24:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:04 smithi196 ceph-mon[76349]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:24:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:06 smithi184 ceph-mon[102844]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:24:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:06 smithi184 ceph-mon[100622]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:24:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:06 smithi196 ceph-mon[76349]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:24:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:08 smithi184 ceph-mon[102844]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T20:24:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:08 smithi184 ceph-mon[100622]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T20:24:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:09 smithi196 ceph-mon[76349]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T20:24:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:10.206Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:10.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:10.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:11 smithi184 ceph-mon[100622]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:11 smithi184 ceph-mon[102844]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:11 smithi196 ceph-mon[76349]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:24:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:13 smithi184 ceph-mon[100622]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:24:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:13 smithi184 ceph-mon[102844]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:24:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:13 smithi196 ceph-mon[76349]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:24:14.013 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:13.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:14.013 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:13.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:14.013 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:13.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[102844]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[100622]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:15 smithi196 ceph-mon[76349]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T20:24:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:17 smithi196 ceph-mon[76349]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[102844]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[100622]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:19.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:19 smithi196 ceph-mon[76349]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T20:24:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:19 smithi184 ceph-mon[100622]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T20:24:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:19 smithi184 ceph-mon[102844]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T20:24:20.075 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:24:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:20.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:20.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:20.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 41m - - 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 40m - - 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (35m) 6s ago 36m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (36m) 4s ago 36m 91.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (29m) 6s ago 34m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (29m) 4s ago 40m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (29m) 6s ago 41m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (28m) 6s ago 42m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:24:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (27m) 4s ago 40m 84.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (28m) 6s ago 40m 92.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (35m) 6s ago 35m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (35m) 4s ago 35m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (26m) 6s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (25m) 6s ago 39m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (25m) 6s ago 39m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (24m) 6s ago 38m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (23m) 4s ago 38m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (22m) 4s ago 37m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (21m) 4s ago 37m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (20m) 4s ago 37m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (29m) 4s ago 36m 65.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (20m) 6s ago 34m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (20m) 4s ago 34m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (20m) 6s ago 34m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:24:20.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (19m) 4s ago 34m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:20.917 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:24:20.918 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:24:21.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:24:21.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:24:21.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[100622]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:21.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[100622]: from='client.56129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[100622]: from='client.56135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1954303803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:21.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[102844]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:21.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[102844]: from='client.56129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[102844]: from='client.56135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1954303803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:21 smithi196 ceph-mon[76349]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:21 smithi196 ceph-mon[76349]: from='client.56129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:21 smithi196 ceph-mon[76349]: from='client.56135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1954303803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:21.514 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:24:21.514 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:21.514 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:24:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[100622]: from='client.56141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[100622]: from='client.56153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1516580967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[102844]: from='client.56141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[102844]: from='client.56153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1516580967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:22 smithi196 ceph-mon[76349]: from='client.56141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:22 smithi196 ceph-mon[76349]: from='client.56153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1516580967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:23 smithi184 ceph-mon[100622]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:24:23.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:23 smithi184 ceph-mon[102844]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:24:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:23 smithi196 ceph-mon[76349]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:24:23.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:23.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:23.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:23.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:23.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:23.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:25 smithi196 ceph-mon[76349]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:24:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:25 smithi184 ceph-mon[100622]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:24:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:25 smithi184 ceph-mon[102844]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:24:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:27 smithi196 ceph-mon[76349]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:27 smithi184 ceph-mon[100622]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:27 smithi184 ceph-mon[102844]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T20:24:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:29 smithi196 ceph-mon[76349]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:24:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:29 smithi184 ceph-mon[100622]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:24:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:29.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:29 smithi184 ceph-mon[102844]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:24:29.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:30.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:30.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:30.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:31 smithi196 ceph-mon[76349]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:24:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:31 smithi184 ceph-mon[100622]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:24:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:31 smithi184 ceph-mon[102844]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:24:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:24:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:33 smithi196 ceph-mon[76349]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:24:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:33 smithi184 ceph-mon[100622]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:24:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:33 smithi184 ceph-mon[102844]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T20:24:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:33.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:33.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:33.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[100622]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[102844]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:35 smithi196 ceph-mon[76349]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:24:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[102844]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[100622]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:37 smithi196 ceph-mon[76349]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T20:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:38.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:39 smithi184 ceph-mon[100622]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-20T20:24:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:39 smithi184 ceph-mon[102844]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-20T20:24:39.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:39 smithi196 ceph-mon[76349]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-20T20:24:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:40.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:40.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:40.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:41 smithi184 ceph-mon[100622]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:24:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:41 smithi184 ceph-mon[102844]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:24:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:41 smithi196 ceph-mon[76349]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:24:42.195 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:43 smithi184 ceph-mon[100622]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:43 smithi184 ceph-mon[102844]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:43 smithi196 ceph-mon[76349]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:44.023 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:43.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:44.023 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:43.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:44.023 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:43.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:45 smithi184 ceph-mon[100622]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:45 smithi184 ceph-mon[102844]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:45 smithi196 ceph-mon[76349]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:47 smithi184 ceph-mon[100622]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:47 smithi184 ceph-mon[102844]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:47 smithi196 ceph-mon[76349]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:49 smithi184 ceph-mon[100622]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:49 smithi184 ceph-mon[102844]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:49 smithi196 ceph-mon[76349]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:50.209Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:50.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:50.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:51 smithi184 ceph-mon[100622]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:51 smithi184 ceph-mon[102844]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:51 smithi196 ceph-mon[76349]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:51.811 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:24:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:24:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:24:52.343 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:24:52.343 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 41m - - 2024-09-20T20:24:52.343 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 41m - - 2024-09-20T20:24:52.343 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (35m) 17s ago 36m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (36m) 16s ago 36m 91.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (29m) 17s ago 34m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (29m) 16s ago 40m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (30m) 17s ago 42m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (28m) 17s ago 42m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (28m) 16s ago 41m 82.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (28m) 17s ago 41m 90.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (36m) 17s ago 36m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (36m) 16s ago 36m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (27m) 17s ago 40m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (26m) 17s ago 40m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (25m) 17s ago 39m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (25m) 17s ago 39m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (23m) 16s ago 38m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:24:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (22m) 16s ago 38m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (21m) 16s ago 38m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (21m) 16s ago 37m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (30m) 16s ago 37m 65.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (20m) 17s ago 35m 86.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (20m) 16s ago 35m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (20m) 17s ago 34m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:24:52.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (20m) 16s ago 34m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:24:52.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:24:52.654 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:24:52.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:24:52.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:24:52.654 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:24:52.925 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:24:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:24:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:24:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:24:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:24:52.926 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:24:53.235 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:24:53.235 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:53.235 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:53.235 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:24:53.235 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: from='client.56165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: from='client.56171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: from='client.46212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1218837208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2343331070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: from='client.56165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: from='client.56171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: from='client.46212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1218837208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2343331070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: from='client.56165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: from='client.56171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: from='client.46212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1218837208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2343331070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:53.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:53.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:24:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:24:53.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:24:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:54 smithi184 ceph-mon[100622]: from='client.56189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:54 smithi184 ceph-mon[102844]: from='client.56189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:54 smithi196 ceph-mon[76349]: from='client.56189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:55 smithi196 ceph-mon[76349]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[100622]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[102844]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:57 smithi196 ceph-mon[76349]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[100622]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[102844]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:24:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:59 smithi196 ceph-mon[76349]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:59 smithi184 ceph-mon[100622]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:59 smithi184 ceph-mon[102844]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:00.209Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:00.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:00.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:01.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:01 smithi196 ceph-mon[76349]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:01 smithi184 ceph-mon[100622]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:01 smithi184 ceph-mon[102844]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:03 smithi196 ceph-mon[76349]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:03.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:03 smithi184 ceph-mon[100622]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:03.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:03 smithi184 ceph-mon[102844]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:03.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:03.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:03.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:05 smithi196 ceph-mon[76349]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:05 smithi184 ceph-mon[100622]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:05 smithi184 ceph-mon[102844]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:07 smithi196 ceph-mon[76349]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:07 smithi184 ceph-mon[102844]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:07 smithi184 ceph-mon[100622]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:09 smithi196 ceph-mon[76349]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:09 smithi184 ceph-mon[102844]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:09 smithi184 ceph-mon[100622]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:10.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:10.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:10.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:11 smithi196 ceph-mon[76349]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:11 smithi184 ceph-mon[100622]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:11 smithi184 ceph-mon[102844]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:25:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:13 smithi196 ceph-mon[76349]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:13.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:13 smithi184 ceph-mon[100622]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:13 smithi184 ceph-mon[102844]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:13.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:13.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:13.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[102844]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[100622]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:15 smithi196 ceph-mon[76349]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:15.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[100622]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[102844]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:17 smithi196 ceph-mon[76349]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:19.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:19 smithi196 ceph-mon[76349]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:19 smithi184 ceph-mon[100622]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:19 smithi184 ceph-mon[102844]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:20.211Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:20.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:20.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:21.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:21 smithi196 ceph-mon[76349]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:21 smithi184 ceph-mon[100622]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:21 smithi184 ceph-mon[102844]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:25:23.531 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:25:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:23 smithi184 ceph-mon[100622]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:23 smithi184 ceph-mon[102844]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:23.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:23.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:23.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:23 smithi196 ceph-mon[76349]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 42m - - 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 41m - - 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (36m) 9s ago 37m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (37m) 7s ago 37m 90.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (30m) 9s ago 35m 62.6M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (30m) 7s ago 41m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (31m) 9s ago 42m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (29m) 9s ago 43m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (28m) 7s ago 41m 83.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (29m) 9s ago 41m 91.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (36m) 9s ago 36m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (36m) 7s ago 36m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:25:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (27m) 9s ago 40m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (26m) 9s ago 40m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (26m) 9s ago 40m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (25m) 9s ago 39m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (24m) 7s ago 39m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (23m) 7s ago 39m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (22m) 7s ago 38m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (21m) 7s ago 38m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (30m) 7s ago 37m 63.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (21m) 9s ago 35m 86.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (21m) 7s ago 35m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (21m) 9s ago 35m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:25:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (21m) 7s ago 35m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:25:24.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:25:24.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:25:24.684 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:25:24.685 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:25:24.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:25:24.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:24 smithi184 ceph-mon[100622]: from='client.46236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:24.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2718897189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:24.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:24 smithi184 ceph-mon[102844]: from='client.46236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:24.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2718897189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:24 smithi196 ceph-mon[76349]: from='client.46236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2718897189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:25.020 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:25:25.020 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:25.020 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:25.020 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:25:25.020 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:25:25.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:25 smithi196 ceph-mon[76349]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:25 smithi196 ceph-mon[76349]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:25.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:25 smithi196 ceph-mon[76349]: from='client.56213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4280981974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[100622]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[100622]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[100622]: from='client.56213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4280981974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[102844]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[102844]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[102844]: from='client.56213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4280981974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:26.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:26 smithi184 ceph-mon[100622]: from='client.46260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:26.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:26 smithi184 ceph-mon[102844]: from='client.46260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:26 smithi196 ceph-mon[76349]: from='client.46260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:27 smithi184 ceph-mon[100622]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:27 smithi184 ceph-mon[102844]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:27 smithi196 ceph-mon[76349]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:29 smithi184 ceph-mon[100622]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:29 smithi184 ceph-mon[102844]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:29 smithi196 ceph-mon[76349]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:30.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:30.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:30.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:31 smithi184 ceph-mon[100622]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:31 smithi184 ceph-mon[102844]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:31 smithi196 ceph-mon[76349]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:25:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:33 smithi184 ceph-mon[100622]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:33 smithi184 ceph-mon[102844]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:33.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:33.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:33.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:33 smithi196 ceph-mon[76349]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[102844]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[100622]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:35 smithi196 ceph-mon[76349]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[102844]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[100622]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:37 smithi196 ceph-mon[76349]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:38.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:39 smithi184 ceph-mon[100622]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:39 smithi184 ceph-mon[102844]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:39 smithi196 ceph-mon[76349]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:40.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:40.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:40.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:41 smithi184 ceph-mon[100622]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:41 smithi184 ceph-mon[102844]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:41.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:41 smithi196 ceph-mon[76349]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:25:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:43 smithi184 ceph-mon[100622]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:43 smithi184 ceph-mon[102844]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:43.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:43.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:43.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:43 smithi196 ceph-mon[76349]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:44.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:44.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:45 smithi196 ceph-mon[76349]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:45.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:45 smithi184 ceph-mon[100622]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:45.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:45 smithi184 ceph-mon[102844]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:47 smithi196 ceph-mon[76349]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:47 smithi184 ceph-mon[100622]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:47 smithi184 ceph-mon[102844]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:49 smithi196 ceph-mon[76349]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:49 smithi184 ceph-mon[100622]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:49 smithi184 ceph-mon[102844]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:50.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:50.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:50.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:51 smithi196 ceph-mon[76349]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:51 smithi184 ceph-mon[100622]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:51 smithi184 ceph-mon[102844]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:25:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:25:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:53 smithi196 ceph-mon[76349]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:53 smithi184 ceph-mon[100622]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:53 smithi184 ceph-mon[102844]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:53.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:53.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:25:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:25:53.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:25:55.311 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 42m - - 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 42m - - 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (36m) 0s ago 37m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (37m) 19s ago 37m 90.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:25:55.836 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (30m) 0s ago 35m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (30m) 19s ago 41m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (31m) 0s ago 43m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (29m) 0s ago 43m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (29m) 19s ago 42m 82.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (29m) 0s ago 42m 91.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:25:55.837 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (37m) 0s ago 37m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (37m) 19s ago 37m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (28m) 0s ago 41m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (27m) 0s ago 41m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (26m) 0s ago 40m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (26m) 0s ago 40m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (24m) 19s ago 39m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:25:55.838 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (24m) 19s ago 39m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (22m) 19s ago 39m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (22m) 19s ago 38m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (31m) 19s ago 38m 63.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (21m) 0s ago 36m 86.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (21m) 19s ago 36m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:25:55.839 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (21m) 0s ago 36m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:25:55.840 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (21m) 19s ago 35m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:25:55.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[100622]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:55.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[102844]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:55.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:55 smithi196 ceph-mon[76349]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:56.148 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:25:56.148 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:25:56.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:25:56.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:25:56.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:25:56.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:25:56.424 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:25:56.425 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:25:56.425 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:25:56.736 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:25:56.736 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:56.737 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:56.737 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:25:56.737 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:25:56.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[100622]: from='client.56237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[100622]: from='client.56243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/288619510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:56.737 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[102844]: from='client.56237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.737 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[102844]: from='client.56243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/288619510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:56 smithi196 ceph-mon[76349]: from='client.56237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:56 smithi196 ceph-mon[76349]: from='client.56243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/288619510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='client.46284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='client.56261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1213468186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='client.46284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='client.56261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1213468186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='client.46284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='client.56261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1213468186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:25:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:59 smithi196 ceph-mon[76349]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:25:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:59 smithi184 ceph-mon[100622]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:59 smithi184 ceph-mon[102844]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:00.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:00.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:00.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:01 smithi196 ceph-mon[76349]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:01 smithi184 ceph-mon[100622]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:01 smithi184 ceph-mon[102844]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:26:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:03 smithi196 ceph-mon[76349]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:03 smithi184 ceph-mon[100622]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:03 smithi184 ceph-mon[102844]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:03.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:03.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:03.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:05.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:05 smithi184 ceph-mon[100622]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:05.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:05 smithi184 ceph-mon[102844]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:05 smithi196 ceph-mon[76349]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:07 smithi196 ceph-mon[76349]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:07 smithi184 ceph-mon[100622]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:07 smithi184 ceph-mon[102844]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:09 smithi196 ceph-mon[76349]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:09 smithi184 ceph-mon[100622]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:09 smithi184 ceph-mon[102844]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:10.214Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:10.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:10.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:10 smithi196 ceph-mon[76349]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:10 smithi184 ceph-mon[100622]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:10 smithi184 ceph-mon[102844]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:26:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:13 smithi196 ceph-mon[76349]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:13 smithi184 ceph-mon[100622]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:13 smithi184 ceph-mon[102844]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:13.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:13.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:13.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:15 smithi196 ceph-mon[76349]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:15 smithi184 ceph-mon[100622]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:15 smithi184 ceph-mon[102844]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[100622]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[102844]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:17 smithi196 ceph-mon[76349]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:19.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:19 smithi184 ceph-mon[100622]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:19.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:19 smithi184 ceph-mon[102844]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:19 smithi196 ceph-mon[76349]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:20.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:20.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:20.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:21.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:21 smithi196 ceph-mon[76349]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:21 smithi184 ceph-mon[100622]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:21 smithi184 ceph-mon[102844]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:26:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:23 smithi184 ceph-mon[100622]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:23 smithi184 ceph-mon[102844]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:23 smithi196 ceph-mon[76349]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:23.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:23.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:23.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:25 smithi196 ceph-mon[76349]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:25 smithi184 ceph-mon[100622]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:25 smithi184 ceph-mon[102844]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:27.033 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:26:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:26 smithi184 ceph-mon[100622]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:26 smithi184 ceph-mon[102844]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:26 smithi196 ceph-mon[76349]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 43m - - 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 42m - - 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (37m) 12s ago 38m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (38m) 10s ago 38m 90.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (31m) 12s ago 36m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (31m) 10s ago 42m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (32m) 12s ago 44m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (30m) 12s ago 44m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (29m) 10s ago 42m 83.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (30m) 12s ago 42m 91.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:26:27.622 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (38m) 12s ago 38m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (37m) 10s ago 37m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (28m) 12s ago 41m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (27m) 12s ago 41m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (27m) 12s ago 41m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (26m) 12s ago 40m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (25m) 10s ago 40m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (24m) 10s ago 40m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (23m) 10s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (22m) 10s ago 39m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (31m) 10s ago 38m 66.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (22m) 12s ago 36m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (22m) 10s ago 37m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (22m) 12s ago 36m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:26:27.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (22m) 10s ago 36m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:26:27.946 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[100622]: from='client.56273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[100622]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[102844]: from='client.56273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:27 smithi184 ceph-mon[102844]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:26:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:26:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:26:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:26:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:26:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:26:28.233 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:26:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:27 smithi196 ceph-mon[76349]: from='client.56273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:27 smithi196 ceph-mon[76349]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:28.556 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:26:28.556 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:28.556 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:26:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:26:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/951203266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: from='client.56297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2478162661' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/951203266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: from='client.56297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2478162661' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/951203266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: from='client.56297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2478162661' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:30.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:30.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:30.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:31 smithi196 ceph-mon[76349]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:31 smithi184 ceph-mon[100622]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:31 smithi184 ceph-mon[102844]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:26:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:33 smithi196 ceph-mon[76349]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:33 smithi184 ceph-mon[100622]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:33 smithi184 ceph-mon[102844]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:33.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:33.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:33.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:35.324 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:35 smithi184 ceph-mon[100622]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:35.325 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:35 smithi184 ceph-mon[102844]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:35 smithi196 ceph-mon[76349]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:36.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:37 smithi196 ceph-mon[76349]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[100622]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:37.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.823 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[102844]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:37.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:37.824 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:39 smithi196 ceph-mon[76349]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:39 smithi184 ceph-mon[100622]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:39 smithi184 ceph-mon[102844]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:40.217Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:40.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:40.218Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:41 smithi196 ceph-mon[76349]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:41 smithi184 ceph-mon[100622]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:41 smithi184 ceph-mon[102844]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:26:43.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:43 smithi196 ceph-mon[76349]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:43.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:43 smithi184 ceph-mon[100622]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:43.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:43 smithi184 ceph-mon[102844]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:43.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:43.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:43.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:45.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:45 smithi184 ceph-mon[100622]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:45.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:45 smithi184 ceph-mon[102844]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:45 smithi196 ceph-mon[76349]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:47.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:47 smithi184 ceph-mon[100622]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:47.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:47 smithi184 ceph-mon[102844]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:47 smithi196 ceph-mon[76349]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:49 smithi196 ceph-mon[76349]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:49 smithi184 ceph-mon[100622]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:49 smithi184 ceph-mon[102844]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:26:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:50.218Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:50.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:50.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:51 smithi196 ceph-mon[76349]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:51 smithi184 ceph-mon[100622]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:51 smithi184 ceph-mon[102844]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:26:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:26:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:53 smithi196 ceph-mon[76349]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:26:53.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:53 smithi184 ceph-mon[100622]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:26:53.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:53 smithi184 ceph-mon[102844]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:26:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:53.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:53.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:26:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:26:53.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:26:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:55 smithi196 ceph-mon[76349]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:55 smithi184 ceph-mon[100622]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:55 smithi184 ceph-mon[102844]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[100622]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[102844]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:57 smithi196 ceph-mon[76349]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:26:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:58.854 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:26:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:59.383 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 43m - - 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 43m - - 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (37m) 3s ago 38m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (38m) 2s ago 38m 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (31m) 3s ago 36m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (31m) 2s ago 42m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (32m) 3s ago 44m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (30m) 3s ago 44m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (30m) 2s ago 43m 85.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (30m) 3s ago 43m 93.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:26:59.384 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (38m) 3s ago 38m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (38m) 2s ago 38m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (29m) 3s ago 42m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (28m) 3s ago 42m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (27m) 3s ago 41m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (27m) 3s ago 41m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (25m) 2s ago 40m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (25m) 2s ago 40m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (23m) 2s ago 40m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (23m) 2s ago 39m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (32m) 2s ago 39m 66.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (22m) 3s ago 37m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (22m) 2s ago 37m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (22m) 3s ago 37m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:26:59.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (22m) 2s ago 37m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:26:59.697 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:26:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:26:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:26:59.699 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:26:59.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:59 smithi184 ceph-mon[100622]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:59.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:59.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:59 smithi184 ceph-mon[102844]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:59.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:59 smithi196 ceph-mon[76349]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:59.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:26:59.967 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:26:59.968 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:26:59.969 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:26:59.969 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:26:59.969 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:27:00.280 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:27:00.280 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:00.280 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:00.280 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:27:00.280 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:27:00.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:00.219Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:00.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:00.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:00.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:00.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[100622]: from='client.46344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[100622]: from='client.56315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[100622]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3330452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1838537043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[102844]: from='client.46344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[102844]: from='client.56315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[102844]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3330452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1838537043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:00 smithi196 ceph-mon[76349]: from='client.46344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:00 smithi196 ceph-mon[76349]: from='client.56315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:00 smithi196 ceph-mon[76349]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3330452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1838537043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:01 smithi184 ceph-mon[100622]: from='client.56333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:01 smithi184 ceph-mon[100622]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:01 smithi184 ceph-mon[102844]: from='client.56333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:01 smithi184 ceph-mon[102844]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:01 smithi196 ceph-mon[76349]: from='client.56333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:01 smithi196 ceph-mon[76349]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:27:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:03 smithi196 ceph-mon[76349]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:03 smithi184 ceph-mon[100622]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:03 smithi184 ceph-mon[102844]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:03.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:03.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:03.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:05 smithi196 ceph-mon[76349]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:05 smithi184 ceph-mon[100622]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:05 smithi184 ceph-mon[102844]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:07.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:07 smithi184 ceph-mon[100622]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:07.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:07 smithi184 ceph-mon[102844]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:07 smithi196 ceph-mon[76349]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:08.579 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:08.579 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:09 smithi184 ceph-mon[100622]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:09 smithi184 ceph-mon[102844]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:09 smithi196 ceph-mon[76349]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:10.219Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:10.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:10.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:11 smithi184 ceph-mon[100622]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:11 smithi184 ceph-mon[102844]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:11 smithi196 ceph-mon[76349]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:27:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:13 smithi184 ceph-mon[100622]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:13 smithi184 ceph-mon[102844]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:13.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:13.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:13.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:13 smithi196 ceph-mon[76349]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:14.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:15 smithi196 ceph-mon[76349]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:15 smithi184 ceph-mon[100622]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:15 smithi184 ceph-mon[102844]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:16.854 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:17 smithi196 ceph-mon[76349]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:17.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[100622]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[102844]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:19 smithi196 ceph-mon[76349]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:19 smithi184 ceph-mon[100622]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:19 smithi184 ceph-mon[102844]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:20.220Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:20.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:20.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:21 smithi196 ceph-mon[76349]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:21 smithi184 ceph-mon[100622]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:21 smithi184 ceph-mon[102844]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:27:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:23 smithi196 ceph-mon[76349]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:23 smithi184 ceph-mon[100622]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:23 smithi184 ceph-mon[102844]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:23.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:23.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:23.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:25 smithi196 ceph-mon[76349]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:25 smithi184 ceph-mon[100622]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:25 smithi184 ceph-mon[102844]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:27 smithi196 ceph-mon[76349]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:27.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:27 smithi184 ceph-mon[100622]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:27.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:27 smithi184 ceph-mon[102844]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:29 smithi196 ceph-mon[76349]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:29 smithi184 ceph-mon[102844]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:29 smithi184 ceph-mon[100622]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:30.221Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:30.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:30.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:30.578 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:27:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:30 smithi196 ceph-mon[76349]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:30 smithi196 ceph-mon[76349]: from='client.56345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-mon[100622]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-mon[100622]: from='client.56345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-mon[102844]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:30 smithi184 ceph-mon[102844]: from='client.56345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 44m - - 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 43m - - 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (38m) 15s ago 39m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (39m) 13s ago 39m 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (32m) 15s ago 37m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:27:31.108 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (32m) 13s ago 43m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (33m) 15s ago 45m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (31m) 15s ago 45m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (30m) 13s ago 43m 89.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (31m) 15s ago 43m 94.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (39m) 15s ago 39m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (39m) 13s ago 39m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (29m) 15s ago 43m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (28m) 15s ago 42m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (28m) 15s ago 42m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (27m) 15s ago 41m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (26m) 13s ago 41m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (25m) 13s ago 41m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (24m) 13s ago 40m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (23m) 13s ago 40m 68.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:27:31.109 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (32m) 13s ago 39m 66.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:27:31.110 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (23m) 15s ago 37m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:27:31.110 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (23m) 13s ago 38m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:27:31.110 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (23m) 15s ago 37m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:27:31.110 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (23m) 13s ago 37m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:27:31.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:27:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:27:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:27:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:27:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:27:31.424 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:27:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:27:31.698 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:27:31.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[100622]: from='client.56351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[100622]: from='client.56357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/688670640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:31.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[102844]: from='client.56351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[102844]: from='client.56357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/688670640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:31 smithi196 ceph-mon[76349]: from='client.56351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:31 smithi196 ceph-mon[76349]: from='client.56357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/688670640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:32.011 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:27:32.012 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:32.012 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:27:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:27:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-20T20:27:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:32 smithi196 ceph-mon[76349]: from='client.46404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3894593727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:32 smithi196 ceph-mon[76349]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[100622]: from='client.46404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3894593727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[100622]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[102844]: from='client.46404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3894593727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:32 smithi184 ceph-mon[102844]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:33.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:33.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:33.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:35 smithi196 ceph-mon[76349]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:35 smithi184 ceph-mon[100622]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:35 smithi184 ceph-mon[102844]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[102844]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:36 smithi184 ceph-mon[100622]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:36 smithi196 ceph-mon[76349]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:39 smithi196 ceph-mon[76349]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:39 smithi184 ceph-mon[100622]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:39 smithi184 ceph-mon[102844]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:40.222Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:40.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:40.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:41 smithi196 ceph-mon[76349]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:41 smithi184 ceph-mon[100622]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:41 smithi184 ceph-mon[102844]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-20T20:27:43.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:43 smithi184 ceph-mon[100622]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:43.655 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:43 smithi184 ceph-mon[102844]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:43.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:43 smithi196 ceph-mon[76349]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:43.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:43.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:43.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:44.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:45.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:45 smithi196 ceph-mon[76349]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:45 smithi184 ceph-mon[100622]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:45 smithi184 ceph-mon[102844]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:47 smithi184 ceph-mon[100622]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:47 smithi184 ceph-mon[102844]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:47 smithi196 ceph-mon[76349]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:48.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:48.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:49.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:49 smithi184 ceph-mon[100622]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:49.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:49 smithi184 ceph-mon[102844]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:49 smithi196 ceph-mon[76349]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:50.222Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:50.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:50.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:51 smithi184 ceph-mon[100622]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:51 smithi184 ceph-mon[102844]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:51 smithi196 ceph-mon[76349]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:27:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:27:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:53 smithi184 ceph-mon[100622]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:53 smithi184 ceph-mon[102844]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:53.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:53.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:27:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:27:53.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:27:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:53 smithi196 ceph-mon[76349]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:55.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:55 smithi196 ceph-mon[76349]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:55 smithi184 ceph-mon[100622]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:55 smithi184 ceph-mon[102844]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:56 smithi196 ceph-mon[76349]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[100622]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:56 smithi184 ceph-mon[102844]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:27:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:59 smithi184 ceph-mon[100622]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:59 smithi184 ceph-mon[102844]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:59 smithi196 ceph-mon[76349]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:00.224Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:00.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:00.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:01 smithi184 ceph-mon[100622]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:01 smithi184 ceph-mon[102844]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:01 smithi196 ceph-mon[76349]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:28:02.308 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:28:02.861 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 44m - - 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 44m - - 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (38m) 7s ago 40m 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (39m) 5s ago 39m 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (32m) 7s ago 38m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (32m) 5s ago 43m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (33m) 7s ago 45m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (32m) 7s ago 45m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (31m) 5s ago 44m 86.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:28:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (31m) 7s ago 44m 94.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (39m) 7s ago 39m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (39m) 5s ago 39m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (30m) 7s ago 43m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (29m) 7s ago 43m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (28m) 7s ago 42m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (28m) 7s ago 42m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (26m) 5s ago 42m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (26m) 5s ago 41m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (25m) 5s ago 41m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (24m) 5s ago 40m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (33m) 5s ago 40m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (23m) 7s ago 38m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (23m) 5s ago 38m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:28:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (23m) 7s ago 38m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:28:02.864 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (23m) 5s ago 38m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:28:03.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:28:03.183 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:28:03.462 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:28:03.462 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:28:03.462 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:28:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:28:03.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[102844]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:03.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[102844]: from='client.56381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[102844]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3776526461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:03.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[100622]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:03.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[100622]: from='client.56381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[100622]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3776526461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:03.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:03.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:03.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:03.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:03.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:03.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:03.788 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:28:03.788 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:03.788 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:03.788 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:28:03.788 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:28:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:03 smithi196 ceph-mon[76349]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:03 smithi196 ceph-mon[76349]: from='client.56381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:03 smithi196 ceph-mon[76349]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3776526461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[100622]: from='client.56393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[100622]: from='client.46440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1935025445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[102844]: from='client.56393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[102844]: from='client.46440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1935025445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:04.844 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:04 smithi196 ceph-mon[76349]: from='client.56393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:04 smithi196 ceph-mon[76349]: from='client.46440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:04.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1935025445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:05 smithi184 ceph-mon[102844]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:05 smithi184 ceph-mon[100622]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:05 smithi196 ceph-mon[76349]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:07 smithi184 ceph-mon[100622]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:07 smithi184 ceph-mon[102844]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:07 smithi196 ceph-mon[76349]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:09 smithi184 ceph-mon[100622]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:09 smithi184 ceph-mon[102844]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:09 smithi196 ceph-mon[76349]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:10.413 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:10.224Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:10.414 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:10.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:10.414 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:10.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:11 smithi184 ceph-mon[102844]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:11 smithi184 ceph-mon[100622]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:11 smithi196 ceph-mon[76349]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:28:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:13 smithi184 ceph-mon[102844]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:13 smithi184 ceph-mon[100622]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:13.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:13.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:13.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:13.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:13 smithi196 ceph-mon[76349]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:28:13.810890128Z level=info msg="Completed cleanup jobs" duration=11.400257ms 2024-09-20T20:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:15 smithi184 ceph-mon[100622]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:15 smithi184 ceph-mon[102844]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:15 smithi196 ceph-mon[76349]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[100622]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:16 smithi184 ceph-mon[102844]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:16 smithi196 ceph-mon[76349]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:18 smithi184 ceph-mon[100622]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:18 smithi184 ceph-mon[102844]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:18 smithi196 ceph-mon[76349]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:20.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:20.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:20.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:21 smithi196 ceph-mon[76349]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:21 smithi184 ceph-mon[100622]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:21 smithi184 ceph-mon[102844]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:28:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:23 smithi196 ceph-mon[76349]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:23.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:23 smithi184 ceph-mon[100622]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:23 smithi184 ceph-mon[102844]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:23.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:23.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:23.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:25 smithi196 ceph-mon[76349]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:25 smithi184 ceph-mon[100622]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:25 smithi184 ceph-mon[102844]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:27.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:27 smithi196 ceph-mon[76349]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:27 smithi184 ceph-mon[100622]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:27 smithi184 ceph-mon[102844]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:28.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:28.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:29 smithi196 ceph-mon[76349]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:29 smithi184 ceph-mon[100622]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:29 smithi184 ceph-mon[102844]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:30.226Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:30.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:30.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:31 smithi196 ceph-mon[76349]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:31 smithi184 ceph-mon[100622]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:31 smithi184 ceph-mon[102844]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:28:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:33 smithi184 ceph-mon[100622]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:33 smithi184 ceph-mon[102844]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:33 smithi196 ceph-mon[76349]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:33.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:33.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:33.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:34.084 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 45m - - 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 45m - - 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (39m) 18s ago 40m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (40m) 16s ago 40m 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (33m) 18s ago 38m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (33m) 16s ago 44m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:28:34.614 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (34m) 18s ago 46m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (32m) 18s ago 46m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (31m) 16s ago 44m 87.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (32m) 18s ago 44m 95.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (40m) 18s ago 40m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (40m) 16s ago 40m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (30m) 18s ago 44m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (29m) 18s ago 43m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (29m) 18s ago 43m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (28m) 18s ago 42m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (27m) 16s ago 42m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (26m) 16s ago 42m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (25m) 16s ago 41m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (24m) 16s ago 41m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:28:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (33m) 16s ago 40m 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:28:34.616 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (24m) 18s ago 39m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:28:34.616 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (24m) 16s ago 39m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:28:34.616 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (24m) 18s ago 38m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:28:34.616 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (24m) 16s ago 38m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:28:34.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:28:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:28:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:28:34.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:28:34.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:28:35.189 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:28:35.190 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[100622]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[100622]: from='client.46452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[100622]: from='client.56423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[100622]: from='client.56429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1156608797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[102844]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[102844]: from='client.46452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[102844]: from='client.56423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[102844]: from='client.56429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1156608797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:35 smithi196 ceph-mon[76349]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:35 smithi196 ceph-mon[76349]: from='client.46452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:35 smithi196 ceph-mon[76349]: from='client.56423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:35 smithi196 ceph-mon[76349]: from='client.56429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:35.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1156608797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:35.499 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:28:35.499 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:35.499 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:28:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:28:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:36 smithi196 ceph-mon[76349]: from='client.56441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3725171329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[100622]: from='client.56441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3725171329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[102844]: from='client.56441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3725171329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:37 smithi184 ceph-mon[100622]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:37 smithi184 ceph-mon[102844]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:37 smithi196 ceph-mon[76349]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:39 smithi184 ceph-mon[102844]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:39 smithi184 ceph-mon[100622]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:39 smithi196 ceph-mon[76349]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:40.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:40.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:40.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:41 smithi184 ceph-mon[100622]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:41 smithi184 ceph-mon[102844]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:41 smithi196 ceph-mon[76349]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:28:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:43 smithi184 ceph-mon[102844]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:43 smithi184 ceph-mon[100622]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:43 smithi196 ceph-mon[76349]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:43.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:43.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:43.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:45.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:45 smithi196 ceph-mon[76349]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:45 smithi184 ceph-mon[102844]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:45 smithi184 ceph-mon[100622]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:47 smithi184 ceph-mon[100622]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:47 smithi184 ceph-mon[102844]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:47.511 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:47 smithi196 ceph-mon[76349]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:49.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:49 smithi184 ceph-mon[100622]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:49.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:49 smithi184 ceph-mon[102844]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:49 smithi196 ceph-mon[76349]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:50.468 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:50.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:50.468 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:50.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:50.468 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:50.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:51.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:51 smithi184 ceph-mon[100622]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:51 smithi184 ceph-mon[102844]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:51 smithi196 ceph-mon[76349]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:28:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:28:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:53 smithi184 ceph-mon[100622]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:53 smithi184 ceph-mon[102844]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:53 smithi196 ceph-mon[76349]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:53.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:53.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:28:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:28:53.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:28:55.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:55 smithi196 ceph-mon[76349]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:55 smithi184 ceph-mon[100622]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:55 smithi184 ceph-mon[102844]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[100622]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[102844]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:57 smithi196 ceph-mon[76349]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:28:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:59 smithi184 ceph-mon[100622]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:59 smithi184 ceph-mon[102844]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:59 smithi196 ceph-mon[76349]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:00.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:00.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:00.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:01 smithi184 ceph-mon[100622]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:01 smithi184 ceph-mon[102844]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:01 smithi196 ceph-mon[76349]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:29:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:03 smithi184 ceph-mon[100622]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:03 smithi184 ceph-mon[102844]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:03 smithi196 ceph-mon[76349]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:03.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:03.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:03.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:05 smithi184 ceph-mon[100622]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:05 smithi184 ceph-mon[102844]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:05 smithi196 ceph-mon[76349]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:05.793 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 45m - - 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 45m - - 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (40m) 10s ago 41m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (40m) 8s ago 40m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (34m) 10s ago 39m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (33m) 8s ago 44m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (34m) 10s ago 46m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (33m) 10s ago 46m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (32m) 8s ago 45m 88.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:29:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (32m) 10s ago 45m 96.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (40m) 10s ago 40m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (40m) 8s ago 40m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (31m) 10s ago 44m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (30m) 10s ago 44m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (29m) 10s ago 43m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (29m) 10s ago 43m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (28m) 8s ago 43m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (27m) 8s ago 42m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (26m) 8s ago 42m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (25m) 8s ago 41m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (34m) 8s ago 41m 69.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (25m) 10s ago 39m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (24m) 8s ago 39m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (24m) 10s ago 39m 86.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:29:06.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (24m) 8s ago 39m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:29:06.898 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:29:06.899 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:29:07.205 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:29:07.205 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:07.205 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:07.206 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:29:07.206 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:29:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: from='client.46494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: from='client.56465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3798561651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2140341592' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: from='client.46494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: from='client.56465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3798561651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2140341592' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: from='client.46488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: from='client.46494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: from='client.56465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3798561651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2140341592' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:08.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:08 smithi196 ceph-mon[76349]: from='client.56477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:08.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:08 smithi184 ceph-mon[100622]: from='client.56477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:08 smithi184 ceph-mon[102844]: from='client.56477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:09 smithi196 ceph-mon[76349]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:09 smithi184 ceph-mon[100622]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:09 smithi184 ceph-mon[102844]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:10.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:10.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:10.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:10.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:11 smithi184 ceph-mon[100622]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:11 smithi184 ceph-mon[102844]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:11 smithi196 ceph-mon[76349]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:12.229 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:29:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:13 smithi184 ceph-mon[100622]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:13 smithi184 ceph-mon[102844]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:13 smithi196 ceph-mon[76349]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:13.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:13.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:13.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:15 smithi184 ceph-mon[100622]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:15 smithi184 ceph-mon[102844]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:15 smithi196 ceph-mon[76349]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:17 smithi196 ceph-mon[76349]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[102844]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[100622]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:17.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:19 smithi196 ceph-mon[76349]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:19 smithi184 ceph-mon[100622]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:19 smithi184 ceph-mon[102844]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:20.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:20.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:20.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:21 smithi196 ceph-mon[76349]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:21 smithi184 ceph-mon[100622]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:21 smithi184 ceph-mon[102844]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:29:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:23 smithi196 ceph-mon[76349]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:23.662 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:23 smithi184 ceph-mon[100622]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:23.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:23 smithi184 ceph-mon[102844]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:23.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:23.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:23.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:25 smithi196 ceph-mon[76349]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:25 smithi184 ceph-mon[100622]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:25 smithi184 ceph-mon[102844]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:27 smithi196 ceph-mon[76349]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:27 smithi184 ceph-mon[102844]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:27 smithi184 ceph-mon[100622]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:28.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:29 smithi196 ceph-mon[76349]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:29 smithi184 ceph-mon[102844]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:29 smithi184 ceph-mon[100622]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:30.481 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:30.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:30.481 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:30.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:30.481 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:30.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:31 smithi196 ceph-mon[76349]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:31.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:31 smithi184 ceph-mon[102844]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:31.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:31 smithi184 ceph-mon[100622]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:32.256 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:29:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:33 smithi184 ceph-mon[100622]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:29:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:33 smithi184 ceph-mon[102844]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:29:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:33 smithi196 ceph-mon[76349]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:29:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:33.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:33.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:33.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:35 smithi196 ceph-mon[76349]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:35 smithi184 ceph-mon[100622]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:35 smithi184 ceph-mon[102844]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:37.506 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[100622]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[102844]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:37 smithi196 ceph-mon[76349]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.033 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:29:38.033 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 46m - - 2024-09-20T20:29:38.033 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 20s ago 46m - - 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (40m) 1s ago 41m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (41m) 20s ago 41m 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (34m) 1s ago 39m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (34m) 20s ago 45m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (35m) 1s ago 47m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (33m) 1s ago 47m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (33m) 20s ago 45m 89.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (33m) 1s ago 45m 98.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (41m) 1s ago 41m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (41m) 20s ago 41m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (31m) 1s ago 45m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:29:38.034 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (31m) 1s ago 44m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (30m) 1s ago 44m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (29m) 1s ago 43m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (28m) 20s ago 43m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (27m) 20s ago 43m 59.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (26m) 20s ago 42m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (25m) 20s ago 42m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (34m) 20s ago 41m 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (25m) 1s ago 40m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (25m) 20s ago 40m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (25m) 1s ago 39m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:29:38.035 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (25m) 20s ago 39m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:29:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:29:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:29:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:29:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:29:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:29:38.357 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:29:38.631 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:29:38.632 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:29:38.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:38.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='client.56489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:38.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2930078229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='client.56489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2930078229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='client.56489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2930078229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:38.940 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:29:38.940 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:38.940 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:38.940 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:29:38.940 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[102844]: from='client.56495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[102844]: from='client.56501 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[102844]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[102844]: from='client.56513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3580952204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[100622]: from='client.56495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[100622]: from='client.56501 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[100622]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:39.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[100622]: from='client.56513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3580952204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:39 smithi196 ceph-mon[76349]: from='client.56495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:39 smithi196 ceph-mon[76349]: from='client.56501 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:39 smithi196 ceph-mon[76349]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:39 smithi196 ceph-mon[76349]: from='client.56513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3580952204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:40.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:40.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:40.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:41 smithi184 ceph-mon[100622]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:41 smithi184 ceph-mon[102844]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:41 smithi196 ceph-mon[76349]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:29:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:43 smithi184 ceph-mon[100622]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:43 smithi184 ceph-mon[102844]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:43 smithi196 ceph-mon[76349]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:43.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:43.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:43.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:45 smithi184 ceph-mon[100622]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:45 smithi184 ceph-mon[102844]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:45 smithi196 ceph-mon[76349]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:47 smithi196 ceph-mon[76349]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:47 smithi184 ceph-mon[100622]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:47 smithi184 ceph-mon[102844]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:49.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:49 smithi196 ceph-mon[76349]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:49 smithi184 ceph-mon[100622]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:49 smithi184 ceph-mon[102844]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:50.230Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:50.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:50.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:51 smithi184 ceph-mon[100622]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:51 smithi184 ceph-mon[102844]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:51 smithi196 ceph-mon[76349]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:52.141 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:29:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:29:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:53 smithi184 ceph-mon[100622]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:53 smithi184 ceph-mon[102844]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:53.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:53.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:29:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:29:53.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:29:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:53 smithi196 ceph-mon[76349]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:55 smithi184 ceph-mon[100622]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:55 smithi184 ceph-mon[102844]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:55 smithi196 ceph-mon[76349]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:57 smithi196 ceph-mon[76349]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[100622]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[102844]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:58.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:29:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:59 smithi196 ceph-mon[76349]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:29:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:59 smithi184 ceph-mon[100622]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:00.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:59 smithi184 ceph-mon[102844]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:00.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:00.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:00.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:00.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:01 smithi196 ceph-mon[76349]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:01 smithi184 ceph-mon[100622]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:01 smithi184 ceph-mon[102844]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:30:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:03 smithi196 ceph-mon[76349]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:03 smithi184 ceph-mon[102844]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:03 smithi184 ceph-mon[100622]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:03.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:03.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:03.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:05 smithi196 ceph-mon[76349]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:05 smithi184 ceph-mon[100622]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:05 smithi184 ceph-mon[102844]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:07 smithi196 ceph-mon[76349]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:07 smithi184 ceph-mon[100622]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:07 smithi184 ceph-mon[102844]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:09.237 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 46m - - 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 46m - - 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (41m) 13s ago 42m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (41m) 11s ago 41m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (35m) 13s ago 40m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (34m) 11s ago 46m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (35m) 13s ago 47m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (34m) 13s ago 47m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (33m) 11s ago 46m 90.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:30:09.764 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (33m) 13s ago 46m 98.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (41m) 13s ago 41m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (41m) 11s ago 41m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (32m) 13s ago 45m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (31m) 13s ago 45m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (30m) 13s ago 44m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (30m) 13s ago 44m 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (29m) 11s ago 44m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (28m) 11s ago 43m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (27m) 11s ago 43m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (26m) 11s ago 43m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (35m) 11s ago 42m 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (26m) 13s ago 40m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (25m) 11s ago 40m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (25m) 13s ago 40m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:30:09.765 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (25m) 11s ago 40m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:30:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:09 smithi196 ceph-mon[76349]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:09 smithi184 ceph-mon[100622]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:09 smithi184 ceph-mon[102844]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:10.075 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:30:10.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:30:10.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:30:10.076 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:30:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:30:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:30:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:30:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:30:10.346 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:30:10.346 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:10.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:10.346 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:10.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:10.346 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:10.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:10.655 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:10.656 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:10.656 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:30:10.656 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:30:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[100622]: from='client.46560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[100622]: from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3882644947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[102844]: from='client.46560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[102844]: from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3882644947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:10 smithi196 ceph-mon[76349]: from='client.46560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:10 smithi196 ceph-mon[76349]: from='client.46566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3882644947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[100622]: from='client.46572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[100622]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[100622]: from='client.56549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2828094543' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[102844]: from='client.46572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[102844]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[102844]: from='client.56549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2828094543' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:11 smithi196 ceph-mon[76349]: from='client.46572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:11 smithi196 ceph-mon[76349]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:11 smithi196 ceph-mon[76349]: from='client.56549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2828094543' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:30:12.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:12 smithi184 ceph-mon[100622]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:12.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:12 smithi184 ceph-mon[102844]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:12 smithi196 ceph-mon[76349]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:13.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:13.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:13.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:14 smithi184 ceph-mon[100622]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:14 smithi184 ceph-mon[102844]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:14 smithi196 ceph-mon[76349]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[100622]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[102844]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:17 smithi196 ceph-mon[76349]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:17.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:18.409 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[100622]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[102844]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:19 smithi196 ceph-mon[76349]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:20.232Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:20.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:20.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:21 smithi184 ceph-mon[100622]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:21 smithi184 ceph-mon[102844]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:21 smithi196 ceph-mon[76349]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:30:23.666 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:23 smithi184 ceph-mon[100622]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:23.667 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:23 smithi184 ceph-mon[102844]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:23 smithi196 ceph-mon[76349]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:23.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:23.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:23.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:25.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:25 smithi196 ceph-mon[76349]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:25 smithi184 ceph-mon[100622]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:25 smithi184 ceph-mon[102844]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:26 smithi196 ceph-mon[76349]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:26 smithi184 ceph-mon[100622]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:26 smithi184 ceph-mon[102844]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:29.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:28 smithi196 ceph-mon[76349]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:29.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:28 smithi184 ceph-mon[100622]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:28 smithi184 ceph-mon[102844]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:30.232Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:30.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:30.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:31 smithi196 ceph-mon[76349]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:31 smithi184 ceph-mon[100622]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:31 smithi184 ceph-mon[102844]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:32.150 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-20T20:30:33.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:33 smithi184 ceph-mon[100622]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:33.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:33 smithi184 ceph-mon[102844]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:33 smithi196 ceph-mon[76349]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:33.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:33.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:33.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:35 smithi184 ceph-mon[100622]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:35 smithi184 ceph-mon[102844]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:35 smithi196 ceph-mon[76349]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[100622]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[102844]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:37 smithi196 ceph-mon[76349]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:38.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[100622]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[102844]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:39 smithi196 ceph-mon[76349]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:40.233Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:40.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:40.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:40.954 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 47m - - 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 47m - - 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (41m) 4s ago 42m 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (42m) 2s ago 42m 90.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (35m) 4s ago 40m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (35m) 2s ago 46m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (36m) 4s ago 48m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (34m) 4s ago 48m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:30:41.483 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (34m) 2s ago 46m 88.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (34m) 4s ago 46m 97.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (42m) 4s ago 42m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (42m) 2s ago 42m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (32m) 4s ago 46m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (32m) 4s ago 45m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (31m) 4s ago 45m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (30m) 4s ago 45m 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (29m) 2s ago 44m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (28m) 2s ago 44m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (27m) 2s ago 43m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (26m) 2s ago 43m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (35m) 2s ago 43m 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (26m) 4s ago 41m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:30:41.484 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (26m) 2s ago 41m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:30:41.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (26m) 4s ago 40m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:30:41.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (26m) 2s ago 40m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:30:41.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:41 smithi184 ceph-mon[100622]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:41.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:41 smithi184 ceph-mon[102844]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:41.796 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:30:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:30:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:30:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:30:41.797 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:30:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:41 smithi196 ceph-mon[76349]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-20T20:30:42.066 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:30:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:30:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:30:42.067 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:30:42.377 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:30:42.377 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:42.377 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:42.377 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:30:42.377 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:30:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[102844]: from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[102844]: from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[102844]: from='client.46608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3907010094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/767355693' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[100622]: from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[100622]: from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[100622]: from='client.46608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3907010094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/767355693' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:42 smithi196 ceph-mon[76349]: from='client.46596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:42 smithi196 ceph-mon[76349]: from='client.46602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:42 smithi196 ceph-mon[76349]: from='client.46608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3907010094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/767355693' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-mon[100622]: from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-mon[100622]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-mon[102844]: from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-mon[102844]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:43.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:43.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:43.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:43 smithi196 ceph-mon[76349]: from='client.46620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:43 smithi196 ceph-mon[76349]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:45 smithi184 ceph-mon[100622]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:45 smithi184 ceph-mon[102844]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:45 smithi196 ceph-mon[76349]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:47 smithi196 ceph-mon[76349]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:47 smithi184 ceph-mon[100622]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:47 smithi184 ceph-mon[102844]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:49 smithi196 ceph-mon[76349]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:49 smithi184 ceph-mon[100622]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:49 smithi184 ceph-mon[102844]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:50.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:50.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:50.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:51 smithi196 ceph-mon[76349]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:51 smithi184 ceph-mon[100622]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:51 smithi184 ceph-mon[102844]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:30:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:30:53.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:53 smithi184 ceph-mon[100622]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:53.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:53 smithi184 ceph-mon[102844]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:53 smithi196 ceph-mon[76349]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:53.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:53.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:30:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:30:53.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:30:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:55 smithi196 ceph-mon[76349]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:55 smithi184 ceph-mon[100622]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:55 smithi184 ceph-mon[102844]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:57 smithi196 ceph-mon[76349]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[100622]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[102844]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:59 smithi196 ceph-mon[76349]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:30:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:30:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[100622]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[102844]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:00.235Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:00.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:00.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:01 smithi196 ceph-mon[76349]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:01 smithi184 ceph-mon[100622]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:01 smithi184 ceph-mon[102844]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:02.260 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:31:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:03 smithi196 ceph-mon[76349]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:03 smithi184 ceph-mon[100622]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:03 smithi184 ceph-mon[102844]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:03.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:03.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:03.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:05 smithi196 ceph-mon[76349]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:05 smithi184 ceph-mon[100622]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:05 smithi184 ceph-mon[102844]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:07 smithi196 ceph-mon[76349]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:07 smithi184 ceph-mon[100622]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:07 smithi184 ceph-mon[102844]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:09 smithi196 ceph-mon[76349]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:09 smithi184 ceph-mon[100622]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:09 smithi184 ceph-mon[102844]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:10.236Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:10.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:10.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:11.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:11 smithi196 ceph-mon[76349]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:11 smithi184 ceph-mon[100622]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:11 smithi184 ceph-mon[102844]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:31:12.673 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:31:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:31:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 47m - - 2024-09-20T20:31:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 47m - - 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (42m) 16s ago 43m 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (43m) 14s ago 43m 90.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (36m) 16s ago 41m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (35m) 14s ago 47m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (36m) 16s ago 48m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (35m) 16s ago 48m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (34m) 14s ago 47m 89.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (34m) 16s ago 47m 97.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (42m) 16s ago 42m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (42m) 14s ago 42m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (33m) 16s ago 46m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (32m) 16s ago 46m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (31m) 16s ago 45m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (31m) 16s ago 45m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:31:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (30m) 14s ago 45m 63.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (29m) 14s ago 44m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (28m) 14s ago 44m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (27m) 14s ago 44m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (36m) 14s ago 43m 69.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (27m) 16s ago 41m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (26m) 14s ago 41m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (27m) 16s ago 41m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:31:13.237 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (26m) 14s ago 41m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:13.555 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:31:13.556 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:31:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:31:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:31:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:31:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:31:13.832 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:31:13.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-mon[100622]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:13.832 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1574919571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:13.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-mon[102844]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:13.833 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1574919571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:13.833 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:13.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:13.833 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:13.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:13.833 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:13.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:13 smithi196 ceph-mon[76349]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1574919571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:14.151 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:31:14.151 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:14.151 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:14.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[100622]: from='client.56597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[100622]: from='client.56603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[100622]: from='client.46644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/975035941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[102844]: from='client.56597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[102844]: from='client.56603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[102844]: from='client.46644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:14.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/975035941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:14 smithi196 ceph-mon[76349]: from='client.56597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:14 smithi196 ceph-mon[76349]: from='client.56603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:14 smithi196 ceph-mon[76349]: from='client.46644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/975035941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:15 smithi196 ceph-mon[76349]: from='client.56621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:15 smithi196 ceph-mon[76349]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:16.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:15 smithi184 ceph-mon[100622]: from='client.56621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:16.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:15 smithi184 ceph-mon[100622]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:16.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:15 smithi184 ceph-mon[102844]: from='client.56621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:16.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:15 smithi184 ceph-mon[102844]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:17 smithi196 ceph-mon[76349]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:17.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[100622]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[102844]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:19 smithi196 ceph-mon[76349]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[100622]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[102844]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:20.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:20.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:20.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:21 smithi196 ceph-mon[76349]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:21 smithi184 ceph-mon[100622]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:21 smithi184 ceph-mon[102844]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:31:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:23 smithi196 ceph-mon[76349]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:23 smithi184 ceph-mon[100622]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:23 smithi184 ceph-mon[102844]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:23.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:23.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:23.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:25 smithi196 ceph-mon[76349]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:25 smithi184 ceph-mon[100622]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:25 smithi184 ceph-mon[102844]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:27 smithi196 ceph-mon[76349]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:27 smithi184 ceph-mon[100622]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:28.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:27 smithi184 ceph-mon[102844]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:29 smithi196 ceph-mon[76349]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:29 smithi184 ceph-mon[100622]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:29 smithi184 ceph-mon[102844]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:30.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:30.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:30.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:31.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:31 smithi196 ceph-mon[76349]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:31 smithi184 ceph-mon[100622]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:31 smithi184 ceph-mon[102844]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:31:33.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:33 smithi184 ceph-mon[100622]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:33.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:33 smithi184 ceph-mon[102844]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:33.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:33.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:33.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:33.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:33.746 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:33.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:33 smithi196 ceph-mon[76349]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:35 smithi184 ceph-mon[100622]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:35 smithi184 ceph-mon[102844]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:35 smithi196 ceph-mon[76349]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:37.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:36 smithi184 ceph-mon[100622]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:37.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:36 smithi184 ceph-mon[102844]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:36 smithi196 ceph-mon[76349]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[100622]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[102844]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:39 smithi196 ceph-mon[76349]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:40.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:40.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:40.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:41 smithi184 ceph-mon[100622]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:41 smithi184 ceph-mon[102844]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:41 smithi196 ceph-mon[76349]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:31:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:43 smithi184 ceph-mon[100622]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:43 smithi184 ceph-mon[102844]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:43 smithi196 ceph-mon[76349]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:43.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:43.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:43.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:44.452 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:31:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:44.984 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 48m - - 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 48m - - 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (42m) 7s ago 43m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (43m) 6s ago 43m 92.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (36m) 7s ago 41m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (36m) 6s ago 47m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (37m) 7s ago 49m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (35m) 7s ago 49m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (35m) 6s ago 48m 90.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (35m) 7s ago 47m 99.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (43m) 7s ago 43m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:31:44.985 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (43m) 6s ago 43m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (34m) 7s ago 47m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (33m) 7s ago 46m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (32m) 7s ago 46m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (31m) 7s ago 46m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (30m) 6s ago 45m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (29m) 6s ago 45m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (28m) 6s ago 44m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (28m) 6s ago 44m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (36m) 6s ago 44m 69.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (27m) 7s ago 42m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (27m) 6s ago 42m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (27m) 7s ago 41m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:31:44.986 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (27m) 6s ago 41m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:31:45.298 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:31:45.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:31:45.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:31:45.300 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:31:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:45 smithi184 ceph-mon[100622]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:45 smithi184 ceph-mon[100622]: from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:45 smithi184 ceph-mon[102844]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:45 smithi184 ceph-mon[102844]: from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:45.567 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:31:45.567 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:31:45.567 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:31:45.568 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:31:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:45 smithi196 ceph-mon[76349]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:45 smithi196 ceph-mon[76349]: from='client.46668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:45.875 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:31:45.875 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:45.875 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:45.876 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:45.876 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[100622]: from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[100622]: from='client.46680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3687406540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[100622]: from='client.56657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1473011597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[102844]: from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[102844]: from='client.46680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3687406540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[102844]: from='client.56657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1473011597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:46 smithi196 ceph-mon[76349]: from='client.46674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:46 smithi196 ceph-mon[76349]: from='client.46680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3687406540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:46 smithi196 ceph-mon[76349]: from='client.56657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1473011597' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:47 smithi184 ceph-mon[100622]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:47 smithi184 ceph-mon[102844]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:47 smithi196 ceph-mon[76349]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:49 smithi184 ceph-mon[100622]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:49 smithi184 ceph-mon[102844]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:49 smithi196 ceph-mon[76349]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:50.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:50.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:50.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:51 smithi184 ceph-mon[100622]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:51 smithi184 ceph-mon[102844]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:51 smithi196 ceph-mon[76349]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:31:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:31:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:53 smithi184 ceph-mon[100622]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:53 smithi184 ceph-mon[102844]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:53.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:53 smithi196 ceph-mon[76349]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:53.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:53.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:31:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:31:53.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:31:55.392 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:55 smithi196 ceph-mon[76349]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:55.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:55 smithi184 ceph-mon[100622]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:55.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:55 smithi184 ceph-mon[102844]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:57.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:57 smithi184 ceph-mon[100622]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:57.378 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:57 smithi184 ceph-mon[102844]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:57 smithi196 ceph-mon[76349]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:58.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:59 smithi196 ceph-mon[76349]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[100622]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[102844]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:00.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:00.691 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:00.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:00.692 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:00.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:01 smithi196 ceph-mon[76349]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:01 smithi184 ceph-mon[100622]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:01 smithi184 ceph-mon[102844]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:32:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:03 smithi196 ceph-mon[76349]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:03.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:03 smithi184 ceph-mon[100622]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:03.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:03 smithi184 ceph-mon[102844]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:03.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:03.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:03.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:05 smithi196 ceph-mon[76349]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:05 smithi184 ceph-mon[100622]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:05 smithi184 ceph-mon[102844]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:07 smithi196 ceph-mon[76349]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:07 smithi184 ceph-mon[100622]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:07 smithi184 ceph-mon[102844]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:09 smithi184 ceph-mon[100622]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:09 smithi184 ceph-mon[102844]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:09 smithi196 ceph-mon[76349]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:10.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:10.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:10.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:11.704 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:11 smithi196 ceph-mon[76349]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:11 smithi184 ceph-mon[100622]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:11 smithi184 ceph-mon[102844]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:32:13.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:13 smithi184 ceph-mon[100622]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:13.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:13 smithi184 ceph-mon[102844]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:13.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:13 smithi196 ceph-mon[76349]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:13.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:13.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:13.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:14.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:14.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:15.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:15 smithi184 ceph-mon[100622]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:15.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:15 smithi184 ceph-mon[102844]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:15 smithi196 ceph-mon[76349]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:16.181 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 49m - - 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 48m - - 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (43m) 19s ago 44m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (44m) 17s ago 44m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:32:16.732 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (37m) 19s ago 42m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (37m) 17s ago 48m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (37m) 19s ago 49m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (36m) 19s ago 49m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (35m) 17s ago 48m 88.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (35m) 19s ago 48m 96.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (43m) 19s ago 43m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (43m) 17s ago 43m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (34m) 19s ago 47m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (33m) 19s ago 47m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (33m) 19s ago 47m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (32m) 19s ago 46m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:32:16.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (31m) 17s ago 46m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (30m) 17s ago 45m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (29m) 17s ago 45m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (28m) 17s ago 45m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (37m) 17s ago 44m 69.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (28m) 19s ago 42m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (28m) 17s ago 42m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (28m) 19s ago 42m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:32:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (27m) 17s ago 42m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:32:17.053 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:32:17.053 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:32:17.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:32:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:32:17.328 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:32:17.329 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:32:17.329 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[102844]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[102844]: from='client.56669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[102844]: from='client.56675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/548452590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[100622]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[100622]: from='client.56669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[100622]: from='client.56675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/548452590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:17.692 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:32:17.692 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:17.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:17.692 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:17.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:17 smithi196 ceph-mon[76349]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:17 smithi196 ceph-mon[76349]: from='client.56669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:17 smithi196 ceph-mon[76349]: from='client.56675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/548452590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='client.46716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='client.56693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/19948275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='client.46716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='client.56693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/19948275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='client.46716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='client.56693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/19948275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[100622]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[102844]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:19 smithi196 ceph-mon[76349]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:20.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:20.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:20.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:21 smithi184 ceph-mon[100622]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:21 smithi184 ceph-mon[102844]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:21 smithi196 ceph-mon[76349]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:32:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:23 smithi184 ceph-mon[100622]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:23 smithi184 ceph-mon[102844]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:23.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:23.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:23.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:23 smithi196 ceph-mon[76349]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:25 smithi184 ceph-mon[100622]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:25 smithi184 ceph-mon[102844]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:25 smithi196 ceph-mon[76349]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:27 smithi184 ceph-mon[100622]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:27 smithi184 ceph-mon[102844]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:27.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:27 smithi196 ceph-mon[76349]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:29 smithi184 ceph-mon[100622]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:29 smithi184 ceph-mon[102844]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:29 smithi196 ceph-mon[76349]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:30.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:30.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:30.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:31 smithi196 ceph-mon[76349]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:31 smithi184 ceph-mon[100622]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:31 smithi184 ceph-mon[102844]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:32:33.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:33 smithi196 ceph-mon[76349]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:33 smithi184 ceph-mon[100622]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:33 smithi184 ceph-mon[102844]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:33.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:33.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:33.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:35.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:35 smithi184 ceph-mon[100622]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:35.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:35 smithi184 ceph-mon[102844]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:35 smithi196 ceph-mon[76349]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:37 smithi196 ceph-mon[76349]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:37.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:37 smithi184 ceph-mon[102844]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:37.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:37 smithi184 ceph-mon[100622]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:32:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:38.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[100622]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[102844]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:39 smithi196 ceph-mon[76349]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:40.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:40.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:40.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:41 smithi184 ceph-mon[100622]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:41 smithi184 ceph-mon[102844]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:41 smithi196 ceph-mon[76349]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:32:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:43 smithi184 ceph-mon[100622]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:43 smithi184 ceph-mon[102844]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:43.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:43.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:43.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:44.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:43 smithi196 ceph-mon[76349]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:45 smithi184 ceph-mon[100622]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:45 smithi184 ceph-mon[102844]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:45 smithi196 ceph-mon[76349]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:46 smithi184 ceph-mon[100622]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:46 smithi184 ceph-mon[102844]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:46 smithi196 ceph-mon[76349]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:47.988 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:47.989 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:32:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 49m - - 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 49m - - 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (43m) 10s ago 44m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (44m) 9s ago 44m 90.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (37m) 10s ago 42m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (37m) 9s ago 48m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (38m) 10s ago 50m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:32:48.517 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (36m) 10s ago 50m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (36m) 9s ago 49m 89.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (36m) 10s ago 48m 97.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (44m) 10s ago 44m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (44m) 9s ago 44m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (35m) 10s ago 48m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (34m) 10s ago 47m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (33m) 10s ago 47m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (33m) 10s ago 47m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (31m) 9s ago 46m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (30m) 9s ago 46m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (29m) 9s ago 46m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (29m) 9s ago 45m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (37m) 9s ago 45m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:32:48.518 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (28m) 10s ago 43m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:32:48.519 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (28m) 9s ago 43m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:32:48.519 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (28m) 10s ago 42m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:32:48.519 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (28m) 9s ago 42m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:32:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[102844]: from='client.46740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[102844]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[102844]: from='client.46746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[102844]: from='client.56717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[100622]: from='client.46740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[100622]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[100622]: from='client.46746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:48 smithi184 ceph-mon[100622]: from='client.56717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:48.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:32:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:32:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:32:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:32:48.834 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:32:49.100 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:32:49.101 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:32:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:48 smithi196 ceph-mon[76349]: from='client.46740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:48 smithi196 ceph-mon[76349]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:48 smithi196 ceph-mon[76349]: from='client.46746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:48 smithi196 ceph-mon[76349]: from='client.56717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:32:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:49.415 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:49.415 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/595911775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[100622]: from='client.56729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2515129157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/595911775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[102844]: from='client.56729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2515129157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/595911775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:49 smithi196 ceph-mon[76349]: from='client.56729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2515129157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:50.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:50.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:50.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:50 smithi184 ceph-mon[100622]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:50 smithi184 ceph-mon[102844]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:50 smithi196 ceph-mon[76349]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:32:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:32:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:53 smithi196 ceph-mon[76349]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:53 smithi184 ceph-mon[100622]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:53 smithi184 ceph-mon[102844]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:53.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:53.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:32:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:32:53.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:32:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:55 smithi196 ceph-mon[76349]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:55 smithi184 ceph-mon[100622]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:55 smithi184 ceph-mon[102844]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:57.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:57 smithi184 ceph-mon[100622]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:57.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:57 smithi184 ceph-mon[102844]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:57 smithi196 ceph-mon[76349]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:32:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:59 smithi184 ceph-mon[102844]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:59 smithi184 ceph-mon[100622]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:59 smithi196 ceph-mon[76349]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:00.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:00.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:00.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:00.551 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:00.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:01 smithi184 ceph-mon[100622]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:01 smithi184 ceph-mon[102844]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:01 smithi196 ceph-mon[76349]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:33:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:03 smithi184 ceph-mon[100622]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:33:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:03 smithi184 ceph-mon[102844]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:33:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:03.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:03.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:03.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:03 smithi196 ceph-mon[76349]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T20:33:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:05 smithi184 ceph-mon[100622]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:05 smithi184 ceph-mon[102844]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:05 smithi196 ceph-mon[76349]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:07 smithi184 ceph-mon[100622]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:07 smithi184 ceph-mon[102844]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:07 smithi196 ceph-mon[76349]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:09 smithi184 ceph-mon[100622]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:09 smithi184 ceph-mon[102844]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:09 smithi196 ceph-mon[76349]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:10.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:10.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:10.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:11 smithi184 ceph-mon[100622]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:11 smithi184 ceph-mon[102844]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:11 smithi196 ceph-mon[76349]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:33:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:13 smithi184 ceph-mon[100622]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:13 smithi184 ceph-mon[102844]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:13.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:13.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:13.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:13 smithi196 ceph-mon[76349]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:15.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:15 smithi196 ceph-mon[76349]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:15 smithi184 ceph-mon[100622]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:15 smithi184 ceph-mon[102844]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:17 smithi184 ceph-mon[100622]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:17 smithi184 ceph-mon[102844]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:17 smithi196 ceph-mon[76349]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:19.712 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:33:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:19 smithi184 ceph-mon[100622]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:19 smithi184 ceph-mon[102844]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:19 smithi196 ceph-mon[76349]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 50m - - 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 49m - - 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (44m) 2s ago 45m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (45m) 0s ago 45m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (38m) 2s ago 43m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (38m) 0s ago 49m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (38m) 2s ago 50m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (37m) 2s ago 51m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:33:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (36m) 0s ago 49m 91.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (37m) 2s ago 49m 99.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (44m) 2s ago 44m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (44m) 0s ago 44m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (35m) 2s ago 48m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (34m) 2s ago 48m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (34m) 2s ago 48m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (33m) 2s ago 47m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (32m) 0s ago 47m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (31m) 0s ago 46m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (30m) 0s ago 46m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (29m) 0s ago 46m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (38m) 0s ago 45m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (29m) 2s ago 43m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:33:20.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (29m) 0s ago 43m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:33:20.261 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (29m) 2s ago 43m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:33:20.261 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (28m) 0s ago 43m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:33:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:20.247Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:20.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:20.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:33:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:33:20.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:33:20.860 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:33:20.861 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:33:20.861 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:21.191 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:33:21.191 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:21.191 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:21.191 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:21.191 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: from='client.56741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: from='client.56747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: from='client.56753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3424544037' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/682440471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: from='client.56741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: from='client.56747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: from='client.56753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3424544037' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/682440471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: from='client.56741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: from='client.56747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: from='client.56753 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3424544037' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/682440471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:33:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:22 smithi196 ceph-mon[76349]: from='client.56765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:22 smithi184 ceph-mon[100622]: from='client.56765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:22 smithi184 ceph-mon[102844]: from='client.56765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:23 smithi196 ceph-mon[76349]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:23 smithi184 ceph-mon[100622]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:23 smithi184 ceph-mon[102844]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:23.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:23.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:23.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:25 smithi196 ceph-mon[76349]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:25 smithi184 ceph-mon[100622]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:25 smithi184 ceph-mon[102844]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:27 smithi196 ceph-mon[76349]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:27 smithi184 ceph-mon[100622]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:27 smithi184 ceph-mon[102844]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:29 smithi196 ceph-mon[76349]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:29 smithi184 ceph-mon[100622]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:29 smithi184 ceph-mon[102844]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:30.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:30.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:30.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:30.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:30.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:30.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:31 smithi196 ceph-mon[76349]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:31 smithi184 ceph-mon[100622]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:31 smithi184 ceph-mon[102844]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:33:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:33 smithi184 ceph-mon[100622]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:33 smithi184 ceph-mon[102844]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:33.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:33.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:33.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:33.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:33 smithi196 ceph-mon[76349]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:35 smithi196 ceph-mon[76349]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:35 smithi184 ceph-mon[100622]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:35 smithi184 ceph-mon[102844]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:37.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:37 smithi184 ceph-mon[100622]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:37.852 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:37 smithi184 ceph-mon[102844]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:37 smithi196 ceph-mon[76349]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:38.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:38.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.876 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:38.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:39.692 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:39 smithi184 ceph-mon[100622]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:39.693 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:39 smithi184 ceph-mon[102844]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:39 smithi196 ceph-mon[76349]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:40.249Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:40.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:40.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[100622]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:40 smithi184 ceph-mon[102844]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:40 smithi196 ceph-mon[76349]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:33:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:43 smithi184 ceph-mon[100622]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:43 smithi184 ceph-mon[102844]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:43 smithi196 ceph-mon[76349]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:43.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:43.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:43.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:45 smithi184 ceph-mon[100622]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:45 smithi184 ceph-mon[102844]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:45 smithi196 ceph-mon[76349]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:46 smithi184 ceph-mon[100622]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:46 smithi184 ceph-mon[102844]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:46 smithi196 ceph-mon[76349]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:48 smithi184 ceph-mon[100622]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:48 smithi184 ceph-mon[102844]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:48 smithi196 ceph-mon[76349]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:50.249Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:50.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:50.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:51.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:51 smithi184 ceph-mon[100622]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:51.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:51 smithi184 ceph-mon[102844]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:51.485 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:33:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:51 smithi196 ceph-mon[76349]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:52.012 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:33:52.012 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 50m - - 2024-09-20T20:33:52.012 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 50m - - 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (44m) 13s ago 45m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (45m) 12s ago 45m 92.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (38m) 13s ago 43m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (38m) 12s ago 49m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (39m) 13s ago 51m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (37m) 13s ago 51m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (37m) 12s ago 50m 91.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (37m) 13s ago 50m 99.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (45m) 13s ago 45m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (45m) 12s ago 45m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (36m) 13s ago 49m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (35m) 13s ago 49m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (34m) 13s ago 48m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:33:52.013 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (34m) 13s ago 48m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (32m) 12s ago 47m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (31m) 12s ago 47m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (30m) 12s ago 47m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (30m) 12s ago 46m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (38m) 12s ago 46m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (29m) 13s ago 44m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (29m) 12s ago 44m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (29m) 13s ago 43m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:33:52.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (29m) 12s ago 43m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:33:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:52 smithi184 ceph-mon[100622]: from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:33:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:33:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:52 smithi184 ceph-mon[102844]: from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:33:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:33:52.330 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:33:52.601 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:33:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:33:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:33:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:33:52.602 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:33:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:52 smithi196 ceph-mon[76349]: from='client.46812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:52.912 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:33:52.912 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:52.912 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: from='client.56789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1311242567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: from='client.56801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1024862290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: from='client.56789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1311242567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: from='client.56801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1024862290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: from='client.46818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: from='client.56789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1311242567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: from='client.56801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1024862290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:53.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:53.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:33:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:33:53.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:33:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:55 smithi184 ceph-mon[100622]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:55 smithi184 ceph-mon[102844]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:55 smithi196 ceph-mon[76349]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:57 smithi184 ceph-mon[100622]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:57 smithi184 ceph-mon[102844]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:57 smithi196 ceph-mon[76349]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:58.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:33:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:59 smithi184 ceph-mon[102844]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:59 smithi184 ceph-mon[100622]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:59 smithi196 ceph-mon[76349]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:00.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:00.250Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:00.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:00.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:00.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:00.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[100622]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:00 smithi184 ceph-mon[102844]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:00 smithi196 ceph-mon[76349]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:34:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:03 smithi184 ceph-mon[102844]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:03 smithi184 ceph-mon[100622]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:03 smithi196 ceph-mon[76349]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:03.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:03.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:03.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:05 smithi184 ceph-mon[100622]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:34:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:05 smithi184 ceph-mon[102844]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:34:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:05 smithi196 ceph-mon[76349]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:34:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:07 smithi184 ceph-mon[100622]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:34:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:07 smithi184 ceph-mon[102844]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:34:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:07 smithi196 ceph-mon[76349]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:34:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:09 smithi184 ceph-mon[100622]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:34:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:09 smithi184 ceph-mon[102844]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:09 smithi196 ceph-mon[76349]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:34:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:10.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:10.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:10.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:11 smithi184 ceph-mon[100622]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:11 smithi184 ceph-mon[102844]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:11 smithi196 ceph-mon[76349]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:34:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:13 smithi184 ceph-mon[102844]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:34:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:13 smithi184 ceph-mon[100622]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:34:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:13 smithi196 ceph-mon[76349]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:34:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:13.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:13.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:13.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:15.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:15 smithi196 ceph-mon[76349]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:34:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:15 smithi184 ceph-mon[100622]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:34:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:15 smithi184 ceph-mon[102844]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:34:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:17 smithi184 ceph-mon[100622]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T20:34:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:17 smithi184 ceph-mon[102844]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T20:34:17.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:17 smithi196 ceph-mon[76349]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T20:34:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[100622]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[102844]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:19 smithi196 ceph-mon[76349]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:34:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:20.545 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:20.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:20.545 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:20.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:20.545 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:20.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[100622]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:20 smithi184 ceph-mon[102844]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:34:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:20 smithi196 ceph-mon[76349]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:34:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:34:23.215 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:34:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:23 smithi184 ceph-mon[100622]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:34:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:23 smithi184 ceph-mon[102844]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:34:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:23 smithi196 ceph-mon[76349]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 51m - - 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 50m - - 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (45m) 5s ago 46m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (46m) 3s ago 46m 89.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (39m) 5s ago 44m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (39m) 3s ago 50m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (40m) 5s ago 51m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (38m) 5s ago 52m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (37m) 3s ago 50m 91.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (38m) 5s ago 50m 99.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (45m) 5s ago 45m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:34:23.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (45m) 3s ago 45m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (36m) 5s ago 49m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (35m) 5s ago 49m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (35m) 5s ago 49m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (34m) 5s ago 48m 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (33m) 3s ago 48m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (32m) 3s ago 48m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (31m) 3s ago 47m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (30m) 3s ago 47m 68.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (39m) 3s ago 46m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (30m) 5s ago 44m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (30m) 3s ago 44m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (30m) 5s ago 44m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:34:23.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (30m) 3s ago 44m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:34:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:23.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:23.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:23.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:24.096 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:34:24.096 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:34:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:34:24.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:34:24.375 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:34:24.375 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:34:24.376 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[102844]: from='client.56813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[102844]: from='client.56819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2489697276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[100622]: from='client.56813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[100622]: from='client.56819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.377 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2489697276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:24 smithi196 ceph-mon[76349]: from='client.56813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:24 smithi196 ceph-mon[76349]: from='client.56819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2489697276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:24.686 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:34:24.687 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:24.687 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:24.687 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:24.687 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:25.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:25 smithi196 ceph-mon[76349]: from='client.56825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:25 smithi196 ceph-mon[76349]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T20:34:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:25 smithi196 ceph-mon[76349]: from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/79638608' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[100622]: from='client.56825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[100622]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[100622]: from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/79638608' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[102844]: from='client.56825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[102844]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[102844]: from='client.46872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/79638608' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:27.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:26 smithi184 ceph-mon[100622]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T20:34:27.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:26 smithi184 ceph-mon[102844]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T20:34:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:26 smithi196 ceph-mon[76349]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T20:34:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:28 smithi184 ceph-mon[100622]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:34:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:28 smithi184 ceph-mon[102844]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:28 smithi196 ceph-mon[76349]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:30.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:30.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:30.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:31 smithi184 ceph-mon[100622]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:34:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:31 smithi184 ceph-mon[102844]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:34:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:31 smithi196 ceph-mon[76349]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:34:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:34:33.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:33 smithi184 ceph-mon[100622]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:34:33.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:33 smithi184 ceph-mon[102844]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:34:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:33 smithi196 ceph-mon[76349]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:34:33.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:33.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:33.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:33.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:35 smithi184 ceph-mon[100622]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:35 smithi184 ceph-mon[102844]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:35 smithi196 ceph-mon[76349]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:34:37.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:37 smithi196 ceph-mon[76349]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T20:34:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:37 smithi184 ceph-mon[100622]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T20:34:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:37 smithi184 ceph-mon[102844]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T20:34:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:39.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[100622]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T20:34:39.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.504 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[102844]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T20:34:39.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:39 smithi196 ceph-mon[76349]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T20:34:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:39.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:40.487 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:40.253Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:40.487 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:40.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:40.487 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:40.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:41 smithi196 ceph-mon[76349]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T20:34:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[100622]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:41 smithi184 ceph-mon[102844]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T20:34:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:34:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:43 smithi184 ceph-mon[100622]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:43 smithi184 ceph-mon[102844]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:43 smithi196 ceph-mon[76349]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:43.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:43.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:43.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:45 smithi184 ceph-mon[100622]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:45 smithi184 ceph-mon[102844]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:45 smithi196 ceph-mon[76349]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:47 smithi184 ceph-mon[100622]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:47 smithi184 ceph-mon[102844]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:47 smithi196 ceph-mon[76349]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:49 smithi184 ceph-mon[100622]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:49.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:49 smithi184 ceph-mon[102844]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:49 smithi196 ceph-mon[76349]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:50.253Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:50.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:50.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:51 smithi184 ceph-mon[100622]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:51 smithi184 ceph-mon[102844]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:51 smithi196 ceph-mon[76349]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:34:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:34:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:53 smithi184 ceph-mon[100622]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:53 smithi184 ceph-mon[102844]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:53 smithi196 ceph-mon[76349]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:53.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:53.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:34:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:34:53.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:34:54.981 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:34:55.509 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:34:55.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 51m - - 2024-09-20T20:34:55.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 51m - - 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (45m) 17s ago 46m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (46m) 15s ago 46m 89.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (39m) 17s ago 44m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (39m) 15s ago 50m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (40m) 17s ago 52m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (38m) 17s ago 52m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (38m) 15s ago 51m 92.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (38m) 17s ago 51m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (46m) 17s ago 46m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (46m) 15s ago 46m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (37m) 17s ago 50m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (36m) 17s ago 50m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (35m) 17s ago 49m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:34:55.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (35m) 17s ago 49m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (33m) 15s ago 48m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (33m) 15s ago 48m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (31m) 15s ago 48m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (31m) 15s ago 47m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (40m) 15s ago 47m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (30m) 17s ago 45m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (30m) 15s ago 45m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (30m) 17s ago 45m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:34:55.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (30m) 15s ago 44m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:34:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:55 smithi184 ceph-mon[100622]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:55 smithi184 ceph-mon[102844]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:55 smithi196 ceph-mon[76349]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:55.821 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:34:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:34:56.091 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:34:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:34:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:34:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:34:56.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:34:56.404 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:34:56.404 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:56.404 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[100622]: from='client.56849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[100622]: from='client.46890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[100622]: from='client.46896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1301085091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[102844]: from='client.56849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[102844]: from='client.46890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[102844]: from='client.46896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1301085091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:56 smithi196 ceph-mon[76349]: from='client.56849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:56 smithi196 ceph-mon[76349]: from='client.46890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:56 smithi196 ceph-mon[76349]: from='client.46896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1301085091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[100622]: from='client.46908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[100622]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3614382188' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[102844]: from='client.46908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[102844]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3614382188' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:57.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:57 smithi196 ceph-mon[76349]: from='client.46908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:57.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:57 smithi196 ceph-mon[76349]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:57.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3614382188' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:58.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:59.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:59 smithi196 ceph-mon[76349]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:59.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[100622]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:59.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[102844]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:59.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:34:59.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:00.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:00.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:00.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[102844]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[100622]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:01 smithi196 ceph-mon[76349]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:02.139 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:35:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:03 smithi184 ceph-mon[100622]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:03 smithi184 ceph-mon[102844]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:03 smithi196 ceph-mon[76349]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:03.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:03.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:03.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:05 smithi184 ceph-mon[100622]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:05 smithi184 ceph-mon[102844]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:05 smithi196 ceph-mon[76349]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:07 smithi196 ceph-mon[76349]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:07 smithi184 ceph-mon[100622]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:07 smithi184 ceph-mon[102844]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:09 smithi184 ceph-mon[100622]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:09 smithi184 ceph-mon[102844]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:09 smithi196 ceph-mon[76349]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:10.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:10.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:10.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:11 smithi184 ceph-mon[100622]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:11 smithi184 ceph-mon[102844]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:11 smithi196 ceph-mon[76349]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:35:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:13 smithi184 ceph-mon[100622]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:13 smithi184 ceph-mon[102844]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:13 smithi196 ceph-mon[76349]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:13.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:13.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:13.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:15 smithi184 ceph-mon[102844]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:15 smithi184 ceph-mon[100622]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:15 smithi196 ceph-mon[76349]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:17 smithi184 ceph-mon[100622]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:17 smithi184 ceph-mon[102844]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:17 smithi196 ceph-mon[76349]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[100622]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[102844]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:19 smithi196 ceph-mon[76349]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:20.533 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:20.255Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:20.533 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:20.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:20.533 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:20.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[100622]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[102844]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.580 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:21 smithi196 ceph-mon[76349]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:22.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:35:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:23 smithi196 ceph-mon[76349]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:23.688 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:23 smithi184 ceph-mon[100622]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:23.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:23 smithi184 ceph-mon[102844]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:35:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:23.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:23.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:23.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:25 smithi196 ceph-mon[76349]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:25 smithi184 ceph-mon[100622]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:25 smithi184 ceph-mon[102844]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:26.702 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:35:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:35:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 52m - - 2024-09-20T20:35:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 51m - - 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (46m) 8s ago 47m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (47m) 6s ago 47m 92.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (40m) 8s ago 45m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (40m) 6s ago 51m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (41m) 8s ago 53m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (39m) 8s ago 53m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (38m) 6s ago 51m 92.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (39m) 8s ago 51m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (47m) 8s ago 46m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (46m) 6s ago 46m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (37m) 8s ago 50m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (36m) 8s ago 50m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (36m) 8s ago 50m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (35m) 8s ago 49m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (34m) 6s ago 49m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:35:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (33m) 6s ago 49m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (32m) 6s ago 48m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (31m) 6s ago 48m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (40m) 6s ago 47m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (31m) 8s ago 45m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (31m) 6s ago 46m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (31m) 8s ago 45m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:35:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (31m) 6s ago 45m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:35:27.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:35:27.540 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:35:27.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:27 smithi184 ceph-mon[100622]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:27.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:27 smithi184 ceph-mon[102844]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:27 smithi196 ceph-mon[76349]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:35:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:35:27.812 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:35:28.119 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:35:28.119 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:28.119 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.46920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.56891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.46932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4207906047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1732886014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.46920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.56891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.46932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:28.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4207906047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:28.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1732886014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.46920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.56891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.46932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4207906047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1732886014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:29 smithi196 ceph-mon[76349]: from='client.56909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:29 smithi196 ceph-mon[76349]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[100622]: from='client.56909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[100622]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[102844]: from='client.56909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[102844]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:30.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:30.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:30.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:31 smithi196 ceph-mon[76349]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:31 smithi184 ceph-mon[100622]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:31 smithi184 ceph-mon[102844]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:35:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:33 smithi196 ceph-mon[76349]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:33.688 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:33 smithi184 ceph-mon[100622]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:33.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:33 smithi184 ceph-mon[102844]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:33.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:33.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:33.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:35 smithi196 ceph-mon[76349]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:35 smithi184 ceph-mon[100622]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:35 smithi184 ceph-mon[102844]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:37 smithi184 ceph-mon[100622]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:37 smithi184 ceph-mon[102844]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:37 smithi196 ceph-mon[76349]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:38.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:39 smithi196 ceph-mon[76349]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[100622]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[102844]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:40.257Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:40.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:40.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:41.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[102844]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[100622]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:41 smithi196 ceph-mon[76349]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:35:43.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:43 smithi184 ceph-mon[100622]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:43.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:43 smithi184 ceph-mon[102844]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:43.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:43.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:43.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:43.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:43 smithi196 ceph-mon[76349]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:45 smithi184 ceph-mon[100622]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:45 smithi184 ceph-mon[102844]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:45 smithi196 ceph-mon[76349]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:47 smithi184 ceph-mon[100622]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:47 smithi184 ceph-mon[102844]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:47 smithi196 ceph-mon[76349]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:49 smithi184 ceph-mon[100622]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:49 smithi184 ceph-mon[102844]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:49 smithi196 ceph-mon[76349]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:50.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:50.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:50.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:51 smithi184 ceph-mon[100622]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:51 smithi184 ceph-mon[102844]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:51 smithi196 ceph-mon[76349]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:35:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:35:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:53 smithi184 ceph-mon[100622]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:53 smithi184 ceph-mon[102844]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:53.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:53.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:35:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:35:53.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:35:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:53 smithi196 ceph-mon[76349]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:55 smithi184 ceph-mon[100622]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:55 smithi184 ceph-mon[102844]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:55 smithi196 ceph-mon[76349]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:57 smithi184 ceph-mon[100622]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:57 smithi184 ceph-mon[102844]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:57 smithi196 ceph-mon[76349]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:58.415 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:35:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 52m - - 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 52m - - 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (46m) 0s ago 47m 29.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (47m) 18s ago 47m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (40m) 0s ago 45m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (40m) 18s ago 51m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (41m) 0s ago 53m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (39m) 0s ago 53m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:35:58.951 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (39m) 18s ago 52m 93.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (39m) 0s ago 52m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (47m) 0s ago 47m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (47m) 18s ago 47m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (38m) 0s ago 51m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (37m) 0s ago 51m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (36m) 0s ago 50m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (36m) 0s ago 50m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (34m) 18s ago 49m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (34m) 18s ago 49m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (32m) 18s ago 49m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (32m) 18s ago 48m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (41m) 18s ago 48m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (31m) 0s ago 46m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (31m) 18s ago 46m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:35:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (31m) 0s ago 46m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:35:58.953 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (31m) 18s ago 46m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:35:59.264 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:35:59.265 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:35:59.533 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:35:59.533 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:35:59.533 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:35:59.533 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:35:59.533 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:35:59.534 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:35:59.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:59.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='client.56921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/498718439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='client.56921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:35:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/498718439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:35:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:59.843 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:59.843 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:59.843 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='client.56921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='client.46962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:35:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:35:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/498718439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[102844]: from='client.46968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[102844]: from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3481795102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[100622]: from='client.46968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[100622]: from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3481795102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:00.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:00.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:00.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:00 smithi196 ceph-mon[76349]: from='client.46968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:00 smithi196 ceph-mon[76349]: from='client.46980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3481795102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:01.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:01 smithi196 ceph-mon[76349]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[100622]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[102844]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.876 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:01.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:36:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:03 smithi184 ceph-mon[100622]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:03 smithi184 ceph-mon[102844]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:03.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:03.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:03.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:03 smithi196 ceph-mon[76349]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:05 smithi196 ceph-mon[76349]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:05 smithi184 ceph-mon[102844]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:05 smithi184 ceph-mon[100622]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:07 smithi196 ceph-mon[76349]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:07 smithi184 ceph-mon[100622]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:07 smithi184 ceph-mon[102844]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:09 smithi196 ceph-mon[76349]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:09 smithi184 ceph-mon[100622]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:09 smithi184 ceph-mon[102844]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:10.259Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:10.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:10.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:11 smithi196 ceph-mon[76349]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:11 smithi184 ceph-mon[100622]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:11 smithi184 ceph-mon[102844]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:36:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:13 smithi196 ceph-mon[76349]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:13 smithi184 ceph-mon[102844]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:13 smithi184 ceph-mon[100622]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:13.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:13.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:13.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:15 smithi196 ceph-mon[76349]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:15 smithi184 ceph-mon[100622]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:15 smithi184 ceph-mon[102844]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:17 smithi196 ceph-mon[76349]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:17 smithi184 ceph-mon[100622]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:17 smithi184 ceph-mon[102844]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:19 smithi196 ceph-mon[76349]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[102844]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[100622]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:20.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:20.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:20.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:21 smithi196 ceph-mon[76349]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[100622]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[102844]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-20T20:36:23.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:23 smithi184 ceph-mon[100622]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:23.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:23 smithi184 ceph-mon[102844]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:23.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:23.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:23.948 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:23.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:23.948 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:23.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:23 smithi196 ceph-mon[76349]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:25.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:25 smithi196 ceph-mon[76349]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:25 smithi184 ceph-mon[100622]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:25 smithi184 ceph-mon[102844]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:27 smithi184 ceph-mon[100622]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:27 smithi184 ceph-mon[102844]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:27 smithi196 ceph-mon[76349]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:29 smithi184 ceph-mon[102844]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:29 smithi184 ceph-mon[100622]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:29 smithi196 ceph-mon[76349]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:30.140 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:36:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:30.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:30.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:30.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 53m - - 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 52m - - 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (47m) 11s ago 48m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (48m) 9s ago 48m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (41m) 11s ago 46m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:36:30.666 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (41m) 9s ago 52m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (42m) 11s ago 54m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (40m) 11s ago 54m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (39m) 9s ago 52m 94.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (40m) 11s ago 52m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (48m) 11s ago 48m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (48m) 9s ago 47m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (38m) 11s ago 52m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (37m) 11s ago 51m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (37m) 11s ago 51m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (36m) 11s ago 50m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (35m) 9s ago 50m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (34m) 9s ago 50m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (33m) 9s ago 49m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (32m) 9s ago 49m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:36:30.667 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (41m) 9s ago 48m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:36:30.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (32m) 11s ago 46m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:36:30.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (32m) 9s ago 47m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:36:30.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (32m) 11s ago 46m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:36:30.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (32m) 9s ago 46m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:36:30.980 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:36:30.981 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:36:30.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[100622]: from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:30.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[100622]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:30.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[100622]: from='client.46998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:30.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[100622]: from='client.56969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:30.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[102844]: from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:30.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[102844]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:30.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[102844]: from='client.46998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:30.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:30 smithi184 ceph-mon[102844]: from='client.56969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:30 smithi196 ceph-mon[76349]: from='client.46992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:30 smithi196 ceph-mon[76349]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:30 smithi196 ceph-mon[76349]: from='client.46998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:30 smithi196 ceph-mon[76349]: from='client.56969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:31.249 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:36:31.249 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:36:31.250 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:36:31.563 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:36:31.563 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:31.564 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:31.564 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3683202455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[102844]: from='client.56981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3003816293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3683202455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[100622]: from='client.56981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3003816293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3683202455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:31 smithi196 ceph-mon[76349]: from='client.56981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3003816293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:36:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:32 smithi184 ceph-mon[100622]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:32 smithi184 ceph-mon[102844]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:32 smithi196 ceph-mon[76349]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:33.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:33.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:33.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:35 smithi184 ceph-mon[100622]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:35 smithi184 ceph-mon[102844]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:35 smithi196 ceph-mon[76349]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:37 smithi184 ceph-mon[100622]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:37 smithi184 ceph-mon[102844]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:37 smithi196 ceph-mon[76349]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[100622]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[102844]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:39 smithi196 ceph-mon[76349]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:40.261Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:40.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:40.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:41 smithi196 ceph-mon[76349]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[100622]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[102844]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:36:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:36:43.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:43 smithi184 ceph-mon[100622]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:43.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:43 smithi184 ceph-mon[102844]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:43 smithi196 ceph-mon[76349]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:43.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:43.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:43.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:45.313 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:45 smithi184 ceph-mon[100622]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:45.313 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:45 smithi184 ceph-mon[102844]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:45 smithi196 ceph-mon[76349]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:47 smithi196 ceph-mon[76349]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:47 smithi184 ceph-mon[100622]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:47 smithi184 ceph-mon[102844]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:49 smithi196 ceph-mon[76349]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:49 smithi184 ceph-mon[100622]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:49 smithi184 ceph-mon[102844]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:50.262Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:50.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:50.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:51 smithi196 ceph-mon[76349]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:51 smithi184 ceph-mon[100622]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:51 smithi184 ceph-mon[102844]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:36:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:36:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:53 smithi196 ceph-mon[76349]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:53.693 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:53 smithi184 ceph-mon[100622]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:53.694 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:53 smithi184 ceph-mon[102844]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:53.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:53.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:36:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:36:53.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:36:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:55 smithi196 ceph-mon[76349]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:55.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:55 smithi184 ceph-mon[100622]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:55.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:55 smithi184 ceph-mon[102844]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:57 smithi196 ceph-mon[76349]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:57 smithi184 ceph-mon[102844]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:57 smithi184 ceph-mon[100622]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:59 smithi196 ceph-mon[76349]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:59 smithi184 ceph-mon[100622]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:59 smithi184 ceph-mon[102844]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:00.263Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:00.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:00.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:00.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:01 smithi196 ceph-mon[76349]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:01.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[102844]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[100622]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:01.860 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:37:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:37:02.393 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 53m - - 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 53m - - 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (47m) 3s ago 49m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (48m) 1s ago 48m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (41m) 3s ago 47m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (41m) 1s ago 52m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (42m) 3s ago 54m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (41m) 3s ago 54m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (40m) 1s ago 53m 95.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (40m) 3s ago 53m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (48m) 3s ago 48m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (48m) 1s ago 48m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (39m) 3s ago 52m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:37:02.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (38m) 3s ago 52m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (37m) 3s ago 51m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (37m) 3s ago 51m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (35m) 1s ago 51m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (35m) 1s ago 50m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (33m) 1s ago 50m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (33m) 1s ago 49m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (42m) 1s ago 49m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (32m) 3s ago 47m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (32m) 1s ago 47m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (32m) 3s ago 47m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:37:02.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (32m) 1s ago 47m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:37:02.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:37:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:37:02.996 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:37:02.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:37:03.316 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:37:03.316 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:03.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:03.316 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:03.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:03.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: from='client.56999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: from='client.57005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2464669659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:03.598 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1596698629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:03.694 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: from='client.56999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: from='client.57005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2464669659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1596698629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:03.695 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: from='client.47028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: from='client.56999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: from='client.57005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:03.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2464669659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:03.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1596698629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:03.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:03.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:03.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:03.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:03.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:03.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:04 smithi196 ceph-mon[76349]: from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:04 smithi184 ceph-mon[100622]: from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:04 smithi184 ceph-mon[102844]: from='client.47052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:05 smithi184 ceph-mon[100622]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:05 smithi184 ceph-mon[102844]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:05 smithi196 ceph-mon[76349]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:07 smithi196 ceph-mon[76349]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:07 smithi184 ceph-mon[100622]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:07 smithi184 ceph-mon[102844]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:09 smithi196 ceph-mon[76349]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:09 smithi184 ceph-mon[100622]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:09 smithi184 ceph-mon[102844]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:10.265Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:10.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:10.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:11 smithi196 ceph-mon[76349]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:11 smithi184 ceph-mon[100622]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:11 smithi184 ceph-mon[102844]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:37:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:13 smithi196 ceph-mon[76349]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:13.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:13 smithi184 ceph-mon[100622]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:13.695 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:13 smithi184 ceph-mon[102844]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:13.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:13.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:13.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:15 smithi184 ceph-mon[100622]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:15 smithi184 ceph-mon[102844]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:15 smithi196 ceph-mon[76349]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:17 smithi184 ceph-mon[100622]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:17 smithi184 ceph-mon[102844]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:17 smithi196 ceph-mon[76349]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:19 smithi184 ceph-mon[100622]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:19 smithi184 ceph-mon[102844]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:19 smithi196 ceph-mon[76349]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:20.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:20.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:20.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:20.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[100622]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[102844]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:21 smithi196 ceph-mon[76349]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:21.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:21.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:37:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:23 smithi184 ceph-mon[100622]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:23 smithi184 ceph-mon[102844]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:23.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:23.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:23.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:23.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:23.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:23 smithi196 ceph-mon[76349]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:25.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:25 smithi184 ceph-mon[100622]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:25.733 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:25 smithi184 ceph-mon[102844]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:25 smithi196 ceph-mon[76349]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:27 smithi184 ceph-mon[100622]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:27 smithi184 ceph-mon[102844]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:27 smithi196 ceph-mon[76349]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:29 smithi184 ceph-mon[100622]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:29 smithi184 ceph-mon[102844]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:29 smithi196 ceph-mon[76349]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:30.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:30.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:30.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:31 smithi184 ceph-mon[100622]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:31 smithi184 ceph-mon[102844]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:31 smithi196 ceph-mon[76349]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:37:33.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:33 smithi184 ceph-mon[100622]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:33.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:33 smithi184 ceph-mon[102844]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:33.611 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:37:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:33 smithi196 ceph-mon[76349]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:33.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:33.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:33.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 54m - - 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 54m - - 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (48m) 14s ago 49m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (49m) 13s ago 49m 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (42m) 14s ago 47m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:37:34.136 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (42m) 13s ago 53m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (43m) 14s ago 55m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (41m) 14s ago 55m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (40m) 13s ago 53m 96.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (41m) 14s ago 53m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (49m) 14s ago 49m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (49m) 13s ago 49m 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (39m) 14s ago 53m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (38m) 14s ago 52m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (38m) 14s ago 52m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (37m) 14s ago 51m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (36m) 13s ago 51m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (35m) 13s ago 51m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (34m) 13s ago 50m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:37:34.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (33m) 13s ago 50m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:37:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (42m) 13s ago 49m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:37:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (33m) 14s ago 48m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:37:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (33m) 13s ago 48m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:37:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (33m) 14s ago 47m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:37:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (33m) 13s ago 47m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:37:34.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:37:34.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:37:34.722 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:37:34.723 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:37:34.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:34 smithi184 ceph-mon[100622]: from='client.57029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:34.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1677480412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:34.724 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:34 smithi184 ceph-mon[102844]: from='client.57029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:34.724 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1677480412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:34 smithi196 ceph-mon[76349]: from='client.57029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1677480412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:35.033 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:37:35.033 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:35.034 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:35.034 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[100622]: from='client.57035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[100622]: from='client.57041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[100622]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2094793816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[102844]: from='client.57035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[102844]: from='client.57041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[102844]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2094793816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:35 smithi196 ceph-mon[76349]: from='client.57035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:35 smithi196 ceph-mon[76349]: from='client.57041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:35 smithi196 ceph-mon[76349]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2094793816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:36 smithi184 ceph-mon[102844]: from='client.57053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:36 smithi184 ceph-mon[100622]: from='client.57053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:36 smithi196 ceph-mon[76349]: from='client.57053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:37 smithi184 ceph-mon[100622]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:37 smithi184 ceph-mon[102844]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:37 smithi196 ceph-mon[76349]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:39 smithi184 ceph-mon[102844]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:39 smithi184 ceph-mon[100622]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:39 smithi196 ceph-mon[76349]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:40.267Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:40.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:40.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:41 smithi196 ceph-mon[76349]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[100622]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[102844]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:37:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:37:43.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:43 smithi196 ceph-mon[76349]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:43 smithi184 ceph-mon[100622]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:43 smithi184 ceph-mon[102844]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:43.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:43.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:43.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:44.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:45 smithi184 ceph-mon[100622]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:45 smithi184 ceph-mon[102844]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:45 smithi196 ceph-mon[76349]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:47 smithi184 ceph-mon[100622]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:47 smithi184 ceph-mon[102844]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:47 smithi196 ceph-mon[76349]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:49 smithi184 ceph-mon[100622]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:49 smithi184 ceph-mon[102844]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:49 smithi196 ceph-mon[76349]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:50.269Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:50.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:50.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:51 smithi196 ceph-mon[76349]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:51 smithi184 ceph-mon[100622]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:51 smithi184 ceph-mon[102844]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:37:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:53 smithi196 ceph-mon[76349]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:53.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:53 smithi184 ceph-mon[100622]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:53.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:53 smithi184 ceph-mon[102844]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:53.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:53.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:37:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:37:53.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:37:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:55 smithi196 ceph-mon[76349]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:55 smithi184 ceph-mon[100622]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:55 smithi184 ceph-mon[102844]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:57.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:57 smithi196 ceph-mon[76349]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:57 smithi184 ceph-mon[100622]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:57 smithi184 ceph-mon[102844]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:59 smithi196 ceph-mon[76349]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:59.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:59 smithi184 ceph-mon[100622]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:59.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:59.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:59 smithi184 ceph-mon[102844]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:59.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:00.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:00.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:00.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[100622]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:00 smithi184 ceph-mon[102844]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:00 smithi196 ceph-mon[76349]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:38:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:03 smithi196 ceph-mon[76349]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:03.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:03 smithi184 ceph-mon[100622]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:03.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:03 smithi184 ceph-mon[102844]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:03.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:03.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:03.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:05.320 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:38:05.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:05 smithi196 ceph-mon[76349]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:05.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:05 smithi184 ceph-mon[100622]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:05.712 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:05 smithi184 ceph-mon[102844]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 54m - - 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 54m - - 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (49m) 6s ago 50m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (49m) 4s ago 49m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (43m) 6s ago 48m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (42m) 4s ago 53m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (43m) 6s ago 55m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (42m) 6s ago 55m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (41m) 4s ago 54m 97.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (41m) 6s ago 54m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (49m) 6s ago 49m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (49m) 4s ago 49m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (40m) 6s ago 53m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (39m) 6s ago 53m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (38m) 6s ago 52m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (38m) 6s ago 52m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (37m) 4s ago 52m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (36m) 4s ago 51m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (35m) 4s ago 51m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (34m) 4s ago 50m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (43m) 4s ago 50m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (34m) 6s ago 48m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (33m) 4s ago 48m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (33m) 6s ago 48m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (33m) 4s ago 48m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:38:06.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:38:06.176 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:38:06.454 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:38:06.455 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:38:06.455 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:38:06.455 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:38:06.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:38:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:06 smithi196 ceph-mon[76349]: from='client.47100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:06 smithi196 ceph-mon[76349]: from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3563229732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[100622]: from='client.47100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[100622]: from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3563229732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[102844]: from='client.47100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[102844]: from='client.47106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:06.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3563229732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:06.766 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:38:06.766 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:06.766 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:06.766 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:06.766 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[100622]: from='client.57077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[100622]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[100622]: from='client.57089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2312277580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[102844]: from='client.57077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[102844]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[102844]: from='client.57089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2312277580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:07 smithi196 ceph-mon[76349]: from='client.57077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:07 smithi196 ceph-mon[76349]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:07 smithi196 ceph-mon[76349]: from='client.57089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2312277580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:09 smithi196 ceph-mon[76349]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:09 smithi184 ceph-mon[100622]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:09 smithi184 ceph-mon[102844]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:10.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:10.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:10.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:11 smithi196 ceph-mon[76349]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:11 smithi184 ceph-mon[100622]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:11 smithi184 ceph-mon[102844]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:38:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:13 smithi196 ceph-mon[76349]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:13.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:13 smithi184 ceph-mon[100622]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:13.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:13 smithi184 ceph-mon[102844]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:14.012 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:13.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:14.012 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:13.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:14.012 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:13.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:38:13.812969596Z level=info msg="Completed cleanup jobs" duration=13.021074ms 2024-09-20T20:38:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:15 smithi196 ceph-mon[76349]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:15 smithi184 ceph-mon[100622]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:15 smithi184 ceph-mon[102844]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:17 smithi196 ceph-mon[76349]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:17 smithi184 ceph-mon[100622]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:17 smithi184 ceph-mon[102844]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:19 smithi196 ceph-mon[76349]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:19 smithi184 ceph-mon[100622]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:19 smithi184 ceph-mon[102844]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:20.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:20.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:20.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:20 smithi196 ceph-mon[76349]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[102844]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:20 smithi184 ceph-mon[100622]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:38:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:23 smithi196 ceph-mon[76349]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:23 smithi184 ceph-mon[100622]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:23 smithi184 ceph-mon[102844]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:23.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:23.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:23.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:23.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:25 smithi196 ceph-mon[76349]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:25 smithi184 ceph-mon[100622]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:25 smithi184 ceph-mon[102844]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:27 smithi184 ceph-mon[100622]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:27 smithi184 ceph-mon[102844]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:27 smithi196 ceph-mon[76349]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:29 smithi184 ceph-mon[100622]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:29 smithi184 ceph-mon[102844]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:29 smithi196 ceph-mon[76349]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:30.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:30.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:30.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:31 smithi196 ceph-mon[76349]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:31 smithi184 ceph-mon[100622]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:31 smithi184 ceph-mon[102844]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:38:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:33 smithi196 ceph-mon[76349]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:33.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:33 smithi184 ceph-mon[100622]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:33.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:33 smithi184 ceph-mon[102844]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:33.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:33.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:33.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:35 smithi196 ceph-mon[76349]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:35 smithi184 ceph-mon[102844]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:35 smithi184 ceph-mon[100622]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:37.061 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:38:37.590 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:37 smithi184 ceph-mon[100622]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:37.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:37 smithi184 ceph-mon[102844]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:37.590 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 55m - - 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 55m - - 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (49m) 17s ago 50m 30.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (50m) 16s ago 50m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (43m) 17s ago 48m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (43m) 16s ago 54m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (44m) 17s ago 56m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (42m) 17s ago 56m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (42m) 16s ago 54m 94.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (42m) 17s ago 54m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:38:37.591 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (50m) 17s ago 50m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (50m) 16s ago 50m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (40m) 17s ago 54m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (40m) 17s ago 53m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (39m) 17s ago 53m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (38m) 17s ago 52m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (37m) 16s ago 52m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (36m) 16s ago 52m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (35m) 16s ago 51m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (34m) 16s ago 51m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (43m) 16s ago 50m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (34m) 17s ago 49m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (34m) 16s ago 49m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (34m) 17s ago 48m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:38:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (34m) 16s ago 48m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:38:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:37 smithi196 ceph-mon[76349]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:37.900 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:38:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:38:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:38:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:38:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:38:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:38:37.902 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:38:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:38:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:38:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:38:38.171 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:38:38.484 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:38:38.484 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:38.484 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:38.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[100622]: from='client.47136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[100622]: from='client.57107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[100622]: from='client.47148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2606263873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[102844]: from='client.47136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[102844]: from='client.57107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[102844]: from='client.47148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2606263873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:38 smithi196 ceph-mon[76349]: from='client.47136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:38 smithi196 ceph-mon[76349]: from='client.57107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:38 smithi196 ceph-mon[76349]: from='client.47148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2606263873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:39 smithi196 ceph-mon[76349]: from='client.57125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:39 smithi196 ceph-mon[76349]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1593945625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[100622]: from='client.57125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[100622]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1593945625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[102844]: from='client.57125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[102844]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1593945625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:40.274Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:40.563 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:40.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:40.563 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:40.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[102844]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:40 smithi184 ceph-mon[100622]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:40 smithi196 ceph-mon[76349]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:38:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:38:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:43 smithi196 ceph-mon[76349]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:43.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:43 smithi184 ceph-mon[100622]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:43.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:43 smithi184 ceph-mon[102844]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:43.984 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:43.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:43.984 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:43.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:43.984 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:43.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:45.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:45 smithi196 ceph-mon[76349]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:45 smithi184 ceph-mon[100622]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:45 smithi184 ceph-mon[102844]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:47.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:47 smithi184 ceph-mon[100622]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:47.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:47 smithi184 ceph-mon[102844]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:47 smithi196 ceph-mon[76349]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:49 smithi184 ceph-mon[100622]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:49 smithi184 ceph-mon[102844]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:49 smithi196 ceph-mon[76349]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:50.275Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:50.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:50.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:51 smithi196 ceph-mon[76349]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:51 smithi184 ceph-mon[100622]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:51 smithi184 ceph-mon[102844]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:38:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:38:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:53 smithi196 ceph-mon[76349]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:53 smithi184 ceph-mon[100622]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:53 smithi184 ceph-mon[102844]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:53.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:53.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:38:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:38:53.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:38:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:55 smithi196 ceph-mon[76349]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:55 smithi184 ceph-mon[100622]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:55 smithi184 ceph-mon[102844]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:57 smithi184 ceph-mon[100622]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:57 smithi184 ceph-mon[102844]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:57 smithi196 ceph-mon[76349]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:59 smithi196 ceph-mon[76349]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:59 smithi184 ceph-mon[102844]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:59 smithi184 ceph-mon[100622]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:00.276Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:00.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:00.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:01 smithi196 ceph-mon[76349]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[100622]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:01 smithi184 ceph-mon[102844]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:39:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:03 smithi184 ceph-mon[100622]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:03 smithi184 ceph-mon[102844]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:03.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:03.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:03.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:03 smithi196 ceph-mon[76349]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:05.844 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:05 smithi196 ceph-mon[76349]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:05 smithi184 ceph-mon[100622]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:05 smithi184 ceph-mon[102844]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:07.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:07 smithi184 ceph-mon[100622]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:07.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:07 smithi184 ceph-mon[102844]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:07 smithi196 ceph-mon[76349]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:08.788 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:39:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:09.348 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:39:09.348 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 55m - - 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 55m - - 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (50m) 9s ago 51m 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (50m) 7s ago 50m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (44m) 9s ago 49m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (43m) 7s ago 54m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (44m) 9s ago 56m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (43m) 9s ago 56m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (42m) 7s ago 55m 97.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (42m) 9s ago 55m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (50m) 9s ago 50m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (50m) 7s ago 50m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (41m) 9s ago 54m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (40m) 9s ago 54m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (39m) 9s ago 53m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (39m) 9s ago 53m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:39:09.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (38m) 7s ago 53m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (37m) 7s ago 52m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (36m) 7s ago 52m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (35m) 7s ago 52m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (44m) 7s ago 51m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (35m) 9s ago 49m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (34m) 7s ago 49m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (34m) 9s ago 49m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:39:09.350 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (34m) 7s ago 49m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:39:09.714 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:39:09.714 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:39:09.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:39:09.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:39:09.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:39:09.991 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:39:09.992 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[100622]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[100622]: from='client.47172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[100622]: from='client.57143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[100622]: from='client.57149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2437482392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[102844]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[102844]: from='client.47172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[102844]: from='client.57143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[102844]: from='client.57149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:09.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2437482392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:09 smithi196 ceph-mon[76349]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:09 smithi196 ceph-mon[76349]: from='client.47172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:09 smithi196 ceph-mon[76349]: from='client.57143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:09 smithi196 ceph-mon[76349]: from='client.57149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:10.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2437482392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:10.307 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:39:10.307 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:10.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:10.307 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:10.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:10.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:10.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:10.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[100622]: from='client.47196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[100622]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3961132828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[102844]: from='client.47196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[102844]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3961132828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:10 smithi196 ceph-mon[76349]: from='client.47196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:11.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:10 smithi196 ceph-mon[76349]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:11.201 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3961132828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:39:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:13 smithi196 ceph-mon[76349]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:13.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:13 smithi184 ceph-mon[100622]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:13.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:13 smithi184 ceph-mon[102844]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:13.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:13.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:13.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:15 smithi196 ceph-mon[76349]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:15 smithi184 ceph-mon[100622]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:15 smithi184 ceph-mon[102844]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:17 smithi196 ceph-mon[76349]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:17 smithi184 ceph-mon[100622]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:17 smithi184 ceph-mon[102844]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:19 smithi196 ceph-mon[76349]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:19 smithi184 ceph-mon[100622]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:19 smithi184 ceph-mon[102844]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:20.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:20.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:20.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[100622]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:21 smithi184 ceph-mon[102844]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:21 smithi196 ceph-mon[76349]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T20:39:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:22 smithi196 ceph-mon[76349]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[100622]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:22 smithi184 ceph-mon[102844]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:23.701Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:23.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:23.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:25 smithi196 ceph-mon[76349]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:25 smithi184 ceph-mon[100622]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:25 smithi184 ceph-mon[102844]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:27 smithi196 ceph-mon[76349]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:27 smithi184 ceph-mon[100622]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:27 smithi184 ceph-mon[102844]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:28.665 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:28.665 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:29.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:29 smithi184 ceph-mon[100622]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:29.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:29 smithi184 ceph-mon[102844]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:29 smithi196 ceph-mon[76349]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:30.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:30.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:30.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:30.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:30.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:30.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:31 smithi196 ceph-mon[76349]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:31 smithi184 ceph-mon[100622]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:31 smithi184 ceph-mon[102844]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:39:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:33 smithi196 ceph-mon[76349]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:33.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:33 smithi184 ceph-mon[100622]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:33.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:33 smithi184 ceph-mon[102844]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:33.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:33.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:33.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:35 smithi196 ceph-mon[76349]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:35 smithi184 ceph-mon[100622]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:35 smithi184 ceph-mon[102844]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:37 smithi196 ceph-mon[76349]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:37 smithi184 ceph-mon[100622]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:37 smithi184 ceph-mon[102844]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:39 smithi196 ceph-mon[76349]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:39 smithi184 ceph-mon[100622]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:39 smithi184 ceph-mon[102844]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:40.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:40.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:40.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:40.616 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:39:41.153 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:39:41.153 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 56m - - 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 56m - - 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (50m) 0s ago 51m 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (51m) 19s ago 51m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (44m) 0s ago 49m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (44m) 19s ago 55m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (45m) 0s ago 57m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (43m) 0s ago 57m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (43m) 19s ago 55m 97.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (43m) 0s ago 55m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (51m) 0s ago 51m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (51m) 19s ago 51m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (41m) 0s ago 55m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:39:41.154 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (41m) 0s ago 54m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (40m) 0s ago 54m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (39m) 0s ago 54m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (38m) 19s ago 53m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (37m) 19s ago 53m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (36m) 19s ago 52m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (35m) 19s ago 52m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (44m) 19s ago 52m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (35m) 0s ago 50m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (35m) 19s ago 50m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (35m) 0s ago 49m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:39:41.155 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (35m) 19s ago 49m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:41.467 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:39:41.468 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:39:41.744 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:39:41.745 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:39:41.745 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[100622]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[100622]: from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:41.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[102844]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:41.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:41.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:41 smithi184 ceph-mon[102844]: from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:41 smithi196 ceph-mon[76349]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:42.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:41 smithi196 ceph-mon[76349]: from='client.47208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.058 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:39:42.058 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:42.058 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:42.058 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:42.058 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='client.57179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='client.47220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1356080946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3618431664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='client.57179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='client.47220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1356080946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3618431664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='client.57179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='client.47220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1356080946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:39:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3618431664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:43.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-mon[100622]: from='client.57197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:43.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-mon[100622]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:43.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-mon[102844]: from='client.57197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:43.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-mon[102844]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:43 smithi196 ceph-mon[76349]: from='client.57197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:43 smithi196 ceph-mon[76349]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:43.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:43.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:43.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:45.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:45 smithi196 ceph-mon[76349]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:45 smithi184 ceph-mon[100622]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:45 smithi184 ceph-mon[102844]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:47 smithi184 ceph-mon[100622]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:47 smithi184 ceph-mon[102844]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:47 smithi196 ceph-mon[76349]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:49 smithi184 ceph-mon[100622]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:49 smithi184 ceph-mon[102844]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:49 smithi196 ceph-mon[76349]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:50.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:50.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:50.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:51 smithi184 ceph-mon[100622]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:51 smithi184 ceph-mon[102844]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:51 smithi196 ceph-mon[76349]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:39:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:39:53.706 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:53 smithi184 ceph-mon[100622]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:53.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:53 smithi184 ceph-mon[102844]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:53 smithi196 ceph-mon[76349]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:53.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:53.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:39:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:39:53.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:39:55.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:55 smithi196 ceph-mon[76349]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:55 smithi184 ceph-mon[100622]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:55 smithi184 ceph-mon[102844]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:57 smithi184 ceph-mon[100622]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:57 smithi184 ceph-mon[102844]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:57 smithi196 ceph-mon[76349]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:59 smithi184 ceph-mon[100622]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:59 smithi184 ceph-mon[102844]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:59 smithi196 ceph-mon[76349]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:00.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:39:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:00.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:00.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:00.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[100622]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[102844]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:01 smithi196 ceph-mon[76349]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:03 smithi196 ceph-mon[76349]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:03.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[100622]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:03.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:03.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-mon[102844]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:03.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:03.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:03.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:05 smithi196 ceph-mon[76349]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:05 smithi184 ceph-mon[100622]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:05 smithi184 ceph-mon[102844]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:07.615 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:07 smithi196 ceph-mon[76349]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:07 smithi184 ceph-mon[100622]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:07 smithi184 ceph-mon[102844]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:08.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:09.608 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:09 smithi184 ceph-mon[100622]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:09 smithi184 ceph-mon[102844]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:09 smithi196 ceph-mon[76349]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:10.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:10.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:10.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:11 smithi196 ceph-mon[76349]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:11 smithi184 ceph-mon[100622]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:11 smithi184 ceph-mon[102844]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:40:12.344 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 56m - - 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 56m - - 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (51m) 12s ago 52m 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (52m) 10s ago 52m 92.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (45m) 12s ago 50m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (44m) 10s ago 56m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (45m) 12s ago 57m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (44m) 12s ago 57m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (43m) 10s ago 56m 99.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (43m) 12s ago 56m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:40:12.870 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (51m) 12s ago 51m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (51m) 10s ago 51m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (42m) 12s ago 55m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (41m) 12s ago 55m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (40m) 12s ago 54m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (40m) 12s ago 54m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (39m) 10s ago 54m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (38m) 10s ago 53m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (37m) 10s ago 53m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (36m) 10s ago 53m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (45m) 10s ago 52m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (36m) 12s ago 50m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (35m) 10s ago 50m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:40:12.871 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (36m) 12s ago 50m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:40:12.872 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (35m) 10s ago 50m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:40:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:40:13.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:40:13.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:40:13.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:13.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:40:13.182 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:40:13.454 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:40:13.454 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:40:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[100622]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[100622]: from='client.57209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[100622]: from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/63408993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[102844]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[102844]: from='client.57209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[102844]: from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/63408993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:13 smithi196 ceph-mon[76349]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:13 smithi196 ceph-mon[76349]: from='client.57209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:13 smithi196 ceph-mon[76349]: from='client.47250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/63408993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:13.766 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:13.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:13.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:13.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:14 smithi196 ceph-mon[76349]: from='client.47256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:14 smithi196 ceph-mon[76349]: from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3226437892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[100622]: from='client.47256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[100622]: from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3226437892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[102844]: from='client.47256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[102844]: from='client.47268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3226437892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:15 smithi196 ceph-mon[76349]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:15 smithi184 ceph-mon[100622]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:15 smithi184 ceph-mon[102844]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:17 smithi196 ceph-mon[76349]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:17 smithi184 ceph-mon[100622]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:17 smithi184 ceph-mon[102844]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:19 smithi196 ceph-mon[76349]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:19 smithi184 ceph-mon[100622]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:19 smithi184 ceph-mon[102844]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:20.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:20.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:20.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[100622]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[102844]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:21 smithi196 ceph-mon[76349]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[100622]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-mon[102844]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:23 smithi196 ceph-mon[76349]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:23.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:23.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:23.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:25 smithi196 ceph-mon[76349]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:25 smithi184 ceph-mon[100622]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:25 smithi184 ceph-mon[102844]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:27 smithi196 ceph-mon[76349]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:27 smithi184 ceph-mon[100622]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:27 smithi184 ceph-mon[102844]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:28.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:29.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:29 smithi196 ceph-mon[76349]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:29.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:29 smithi184 ceph-mon[100622]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:29 smithi184 ceph-mon[102844]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:30.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:30.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:30.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:30.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:30.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:30.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:31.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:31 smithi184 ceph-mon[102844]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:31.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:31 smithi184 ceph-mon[100622]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:31 smithi196 ceph-mon[76349]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:40:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:33 smithi184 ceph-mon[100622]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:33.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:33 smithi184 ceph-mon[102844]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:33 smithi196 ceph-mon[76349]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:33.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:33.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:33.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:35 smithi196 ceph-mon[76349]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:35 smithi184 ceph-mon[100622]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:35 smithi184 ceph-mon[102844]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:37 smithi196 ceph-mon[76349]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:37 smithi184 ceph-mon[100622]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:37 smithi184 ceph-mon[102844]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:39 smithi196 ceph-mon[76349]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:39 smithi184 ceph-mon[100622]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:39 smithi184 ceph-mon[102844]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:40.281Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:40.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:40.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:41 smithi196 ceph-mon[76349]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[100622]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[102844]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:40:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[100622]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:43.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[102844]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:43 smithi196 ceph-mon[76349]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:43.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:40:44.061 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:43.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:44.061 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:43.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:44.061 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:43.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:44.061 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:40:44.591 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:40:44.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 57m - - 2024-09-20T20:40:44.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 57m - - 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (51m) 3s ago 52m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (52m) 2s ago 52m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (45m) 3s ago 50m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (45m) 2s ago 56m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (46m) 3s ago 58m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (44m) 3s ago 58m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (44m) 2s ago 57m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (44m) 3s ago 56m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (52m) 3s ago 52m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (52m) 2s ago 52m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (43m) 3s ago 56m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (42m) 3s ago 55m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:40:44.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (41m) 3s ago 55m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (40m) 3s ago 55m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (39m) 2s ago 54m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (38m) 2s ago 54m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (37m) 2s ago 53m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (37m) 2s ago 53m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (45m) 2s ago 53m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (36m) 3s ago 51m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (36m) 2s ago 51m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (36m) 3s ago 50m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:40:44.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (36m) 2s ago 50m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:40:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:44.901 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:40:44.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:40:44.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:40:44.902 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:40:45.173 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:40:45.174 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:40:45.174 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:40:45.174 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:40:45.174 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:40:45.174 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:40:45.484 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:45.484 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:45.484 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:45.484 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:45.484 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[102844]: from='client.47280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[102844]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[102844]: from='client.57251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[102844]: from='client.57257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/56394403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[100622]: from='client.47280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[100622]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[100622]: from='client.57251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[100622]: from='client.57257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/56394403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:45 smithi196 ceph-mon[76349]: from='client.47280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:45 smithi196 ceph-mon[76349]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:45 smithi196 ceph-mon[76349]: from='client.57251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:45 smithi196 ceph-mon[76349]: from='client.57257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/56394403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:46 smithi196 ceph-mon[76349]: from='client.47304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2534701793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:46 smithi184 ceph-mon[100622]: from='client.47304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2534701793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:46 smithi184 ceph-mon[102844]: from='client.47304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2534701793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:47 smithi196 ceph-mon[76349]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:47 smithi184 ceph-mon[100622]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:47 smithi184 ceph-mon[102844]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:49 smithi196 ceph-mon[76349]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:49 smithi184 ceph-mon[100622]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:49 smithi184 ceph-mon[102844]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:50.282Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:50.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:50.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:51 smithi196 ceph-mon[76349]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:51 smithi184 ceph-mon[100622]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:51 smithi184 ceph-mon[102844]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:52.203 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:40:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:40:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:53 smithi196 ceph-mon[76349]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:53 smithi184 ceph-mon[100622]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.712 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:53 smithi184 ceph-mon[102844]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:53.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:53.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:53.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:40:53.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:40:53.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:40:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:55 smithi196 ceph-mon[76349]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:55 smithi184 ceph-mon[100622]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:55 smithi184 ceph-mon[102844]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:57 smithi196 ceph-mon[76349]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:57 smithi184 ceph-mon[100622]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:57 smithi184 ceph-mon[102844]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:59 smithi184 ceph-mon[102844]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:59 smithi184 ceph-mon[100622]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:59 smithi196 ceph-mon[76349]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:00.283Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:00.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:00.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[100622]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[102844]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:01 smithi196 ceph-mon[76349]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[100622]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[102844]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:03.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:03.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:03.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:03 smithi196 ceph-mon[76349]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:05 smithi184 ceph-mon[100622]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:05 smithi184 ceph-mon[102844]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:05 smithi196 ceph-mon[76349]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:07 smithi184 ceph-mon[100622]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:07 smithi184 ceph-mon[102844]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:07 smithi196 ceph-mon[76349]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:08.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:09.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:09 smithi196 ceph-mon[76349]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:09 smithi184 ceph-mon[102844]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:09 smithi184 ceph-mon[100622]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:10.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:10.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:10.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:11.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:11 smithi184 ceph-mon[100622]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:11.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:11 smithi184 ceph-mon[102844]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:11 smithi196 ceph-mon[76349]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:41:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:13 smithi184 ceph-mon[100622]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:13 smithi184 ceph-mon[102844]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:13.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:13.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:13.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:13.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:13 smithi196 ceph-mon[76349]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:15.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:15 smithi184 ceph-mon[100622]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:15.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:15 smithi184 ceph-mon[102844]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:15.783 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:41:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:15 smithi196 ceph-mon[76349]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:16.314 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:41:16.314 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 57m - - 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 57m - - 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (52m) 15s ago 53m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (53m) 13s ago 53m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (46m) 15s ago 51m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (46m) 13s ago 57m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (46m) 15s ago 58m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (45m) 15s ago 58m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (44m) 13s ago 57m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (44m) 15s ago 57m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (52m) 15s ago 52m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (52m) 13s ago 52m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (43m) 15s ago 56m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (42m) 15s ago 56m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:41:16.315 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (41m) 15s ago 56m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (41m) 15s ago 55m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (40m) 13s ago 55m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (39m) 13s ago 54m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (38m) 13s ago 54m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (37m) 13s ago 54m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (46m) 13s ago 53m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (37m) 15s ago 51m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (37m) 13s ago 51m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (37m) 15s ago 51m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:41:16.316 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (36m) 13s ago 51m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:41:16.627 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:41:16.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:41:16.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:41:16.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:16.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:41:16.628 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:41:16.895 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:41:16.895 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:41:16.895 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:41:16.895 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:41:16.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:41:17.206 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:41:17.207 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:17.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:17.207 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:17.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: from='client.47316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: from='client.47322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: from='client.47328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1602458318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2953936398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: from='client.47316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: from='client.47322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: from='client.47328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1602458318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:17.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2953936398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: from='client.47316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: from='client.47322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: from='client.47328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:17.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1602458318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:17.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2953936398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:18 smithi184 ceph-mon[100622]: from='client.57305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:18 smithi184 ceph-mon[102844]: from='client.57305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:18 smithi196 ceph-mon[76349]: from='client.57305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:19 smithi184 ceph-mon[100622]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:19 smithi184 ceph-mon[102844]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:19 smithi196 ceph-mon[76349]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:20.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:20.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:20.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[100622]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[102844]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:21 smithi196 ceph-mon[76349]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:41:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:23 smithi196 ceph-mon[76349]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[100622]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[102844]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:23.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:23.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:23.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:25 smithi196 ceph-mon[76349]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:25 smithi184 ceph-mon[100622]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:25 smithi184 ceph-mon[102844]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:27 smithi196 ceph-mon[76349]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:27 smithi184 ceph-mon[100622]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:27 smithi184 ceph-mon[102844]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:29 smithi196 ceph-mon[76349]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:29 smithi184 ceph-mon[100622]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:29 smithi184 ceph-mon[102844]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:30.285Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:30.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:30.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:31.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:31 smithi196 ceph-mon[76349]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:31 smithi184 ceph-mon[100622]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:31 smithi184 ceph-mon[102844]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:32.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:41:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:33 smithi184 ceph-mon[100622]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:33 smithi184 ceph-mon[102844]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:33.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:33.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:33.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:33 smithi196 ceph-mon[76349]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:35 smithi196 ceph-mon[76349]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:35 smithi184 ceph-mon[100622]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:35 smithi184 ceph-mon[102844]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:37 smithi196 ceph-mon[76349]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:37 smithi184 ceph-mon[100622]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:37 smithi184 ceph-mon[102844]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:39 smithi184 ceph-mon[100622]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:39 smithi184 ceph-mon[102844]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:39 smithi196 ceph-mon[76349]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:40.286Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:40.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:40.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:41 smithi196 ceph-mon[76349]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[102844]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[100622]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[102844]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[100622]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:43.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:43.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:43.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:43 smithi196 ceph-mon[76349]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:41:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:44 smithi184 ceph-mon[100622]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:44 smithi184 ceph-mon[102844]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:44 smithi196 ceph-mon[76349]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:47.504 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:41:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:47 smithi196 ceph-mon[76349]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:47 smithi184 ceph-mon[100622]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:47 smithi184 ceph-mon[102844]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:48.028 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:41:48.028 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 58m - - 2024-09-20T20:41:48.028 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 58m - - 2024-09-20T20:41:48.028 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (52m) 6s ago 53m 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (53m) 5s ago 53m 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (46m) 6s ago 51m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (46m) 5s ago 57m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (47m) 6s ago 59m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (45m) 6s ago 59m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (45m) 5s ago 58m 99.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (45m) 6s ago 57m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (53m) 6s ago 53m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (53m) 5s ago 53m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (44m) 6s ago 57m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (43m) 6s ago 56m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (42m) 6s ago 56m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (42m) 6s ago 56m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (40m) 5s ago 55m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:41:48.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (39m) 5s ago 55m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (38m) 5s ago 55m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (38m) 5s ago 54m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (46m) 5s ago 54m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (37m) 6s ago 52m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (37m) 5s ago 52m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (37m) 6s ago 51m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:41:48.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (37m) 5s ago 51m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:48.344 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:41:48.345 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:41:48.614 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:41:48.615 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:41:48.615 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:41:48.615 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[100622]: from='client.57317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3316761070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[102844]: from='client.57317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:48.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3316761070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:48 smithi196 ceph-mon[76349]: from='client.57317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3316761070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:48.925 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:41:48.925 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:48.925 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:48.926 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:48.926 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:49.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:49 smithi196 ceph-mon[76349]: from='client.57323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:49 smithi196 ceph-mon[76349]: from='client.57329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:49 smithi196 ceph-mon[76349]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:49.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:49 smithi196 ceph-mon[76349]: from='client.57341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/830898330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[100622]: from='client.57323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[100622]: from='client.57329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[100622]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[100622]: from='client.57341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/830898330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[102844]: from='client.57323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[102844]: from='client.57329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[102844]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:49.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[102844]: from='client.57341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:49.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/830898330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:50.286Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:50.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:50.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:51.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:51 smithi196 ceph-mon[76349]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:51 smithi184 ceph-mon[100622]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:51 smithi184 ceph-mon[102844]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:41:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:41:53.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:53 smithi184 ceph-mon[100622]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:53 smithi184 ceph-mon[102844]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:53 smithi196 ceph-mon[76349]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:53.974 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:53.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:53.974 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:53.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:41:53.974 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:41:53.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:41:55.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:55 smithi184 ceph-mon[100622]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:55.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:55 smithi184 ceph-mon[102844]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:55.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:55 smithi196 ceph-mon[76349]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:57 smithi196 ceph-mon[76349]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:57 smithi184 ceph-mon[102844]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:57 smithi184 ceph-mon[100622]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:59 smithi184 ceph-mon[102844]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:59 smithi184 ceph-mon[100622]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:59 smithi196 ceph-mon[76349]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:00.287Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:00.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:00.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[100622]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[102844]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:01 smithi196 ceph-mon[76349]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:01.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:42:03.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[100622]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:03.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[102844]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:03.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:03 smithi196 ceph-mon[76349]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:03.716Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:03.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:03.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:05 smithi184 ceph-mon[102844]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:05 smithi184 ceph-mon[100622]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:05 smithi196 ceph-mon[76349]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:07.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:07 smithi196 ceph-mon[76349]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:07 smithi184 ceph-mon[100622]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:07 smithi184 ceph-mon[102844]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:09 smithi196 ceph-mon[76349]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:09 smithi184 ceph-mon[100622]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:09 smithi184 ceph-mon[102844]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:10.288Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:10.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:10.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:11.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:11 smithi196 ceph-mon[76349]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:11 smithi184 ceph-mon[100622]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:11 smithi184 ceph-mon[102844]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:42:13.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:13 smithi184 ceph-mon[102844]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:13.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:13 smithi184 ceph-mon[100622]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:13.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:13 smithi196 ceph-mon[76349]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:13.983 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:13.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:13.984 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:13.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:13.984 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:13.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:14.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:15 smithi184 ceph-mon[100622]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:15 smithi184 ceph-mon[102844]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:15 smithi196 ceph-mon[76349]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:17 smithi184 ceph-mon[100622]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:17 smithi184 ceph-mon[102844]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:17 smithi196 ceph-mon[76349]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:19.223 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:42:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:19 smithi184 ceph-mon[100622]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:19 smithi184 ceph-mon[102844]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 59m - - 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 58m - - 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (53m) 18s ago 54m 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (54m) 16s ago 54m 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:42:19.755 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (47m) 18s ago 52m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (47m) 16s ago 58m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (47m) 18s ago 59m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (46m) 18s ago 60m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (45m) 16s ago 58m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (46m) 18s ago 58m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (53m) 18s ago 53m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (53m) 16s ago 53m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (44m) 18s ago 57m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (43m) 18s ago 57m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (43m) 18s ago 57m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (42m) 18s ago 56m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (41m) 16s ago 56m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (40m) 16s ago 55m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (39m) 16s ago 55m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (38m) 16s ago 55m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:42:19.756 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (47m) 16s ago 54m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:42:19.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (38m) 18s ago 52m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:42:19.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (38m) 16s ago 52m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:42:19.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (38m) 18s ago 52m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:42:19.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (37m) 16s ago 52m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:42:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:19 smithi196 ceph-mon[76349]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:20.067 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:42:20.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:42:20.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:42:20.069 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:42:20.338 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:42:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:42:20.338 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:42:20.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:42:20.339 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:20.289Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:20.340 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:20.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:20.340 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:20.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:20.647 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:42:20.647 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:20.647 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:20.648 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:20.648 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[100622]: from='client.47388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[100622]: from='client.57359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[100622]: from='client.47400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2585754292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[102844]: from='client.47388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[102844]: from='client.57359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.648 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[102844]: from='client.47400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.649 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2585754292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:20 smithi196 ceph-mon[76349]: from='client.47388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:20 smithi196 ceph-mon[76349]: from='client.57359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:20 smithi196 ceph-mon[76349]: from='client.47400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2585754292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[100622]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[100622]: from='client.57377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2866378390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[102844]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[102844]: from='client.57377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2866378390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:21 smithi196 ceph-mon[76349]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:21 smithi196 ceph-mon[76349]: from='client.57377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2866378390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:22] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[100622]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:23.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[102844]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:23.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:23 smithi196 ceph-mon[76349]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:23.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:23.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:23.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:23.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:23.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:25 smithi184 ceph-mon[102844]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:25 smithi184 ceph-mon[100622]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:25 smithi196 ceph-mon[76349]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:27 smithi196 ceph-mon[76349]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:27 smithi184 ceph-mon[100622]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:27 smithi184 ceph-mon[102844]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:29 smithi196 ceph-mon[76349]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:29 smithi184 ceph-mon[100622]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:29 smithi184 ceph-mon[102844]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:30.290Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:30.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:30.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:31.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:31 smithi196 ceph-mon[76349]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:31 smithi184 ceph-mon[100622]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:31 smithi184 ceph-mon[102844]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:42:33.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:33 smithi184 ceph-mon[100622]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:33.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:33 smithi184 ceph-mon[102844]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:33 smithi196 ceph-mon[76349]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:33.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:33.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:33.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:35.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:35 smithi184 ceph-mon[100622]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:35.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:35 smithi184 ceph-mon[102844]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:35 smithi196 ceph-mon[76349]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:37 smithi184 ceph-mon[100622]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:37 smithi184 ceph-mon[102844]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:37 smithi196 ceph-mon[76349]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:39 smithi184 ceph-mon[100622]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:39 smithi184 ceph-mon[102844]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:39 smithi196 ceph-mon[76349]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:40.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:40.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:40.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:41 smithi184 ceph-mon[100622]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:41 smithi184 ceph-mon[102844]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:41 smithi196 ceph-mon[76349]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:42:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:43 smithi196 ceph-mon[76349]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[100622]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[102844]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:42:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:43.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:43.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:43.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:45 smithi196 ceph-mon[76349]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:45 smithi184 ceph-mon[100622]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:45 smithi184 ceph-mon[102844]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:47 smithi196 ceph-mon[76349]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:47 smithi184 ceph-mon[100622]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:47 smithi184 ceph-mon[102844]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:49 smithi196 ceph-mon[76349]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:49 smithi184 ceph-mon[100622]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:49 smithi184 ceph-mon[102844]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:50.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:50.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:50.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:42:51.472 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:42:51.472 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 59m - - 2024-09-20T20:42:51.472 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 59m - - 2024-09-20T20:42:51.472 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (53m) 9s ago 54m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:42:51.472 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (54m) 8s ago 54m 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (47m) 9s ago 52m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (47m) 8s ago 58m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (48m) 9s ago 60m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (46m) 9s ago 60m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (46m) 8s ago 59m 99.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (46m) 9s ago 59m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (54m) 9s ago 54m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (54m) 8s ago 54m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (45m) 9s ago 58m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (44m) 9s ago 58m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (43m) 9s ago 57m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (43m) 9s ago 57m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (41m) 8s ago 56m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:42:51.473 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (40m) 8s ago 56m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (39m) 8s ago 56m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (39m) 8s ago 55m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (47m) 8s ago 55m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (38m) 9s ago 53m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (38m) 8s ago 53m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (38m) 9s ago 52m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:42:51.474 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (38m) 8s ago 52m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:42:51.786 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:42:51.786 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:42:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:42:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:42:51.788 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:42:51.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:51 smithi184 ceph-mon[100622]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:51.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:51 smithi184 ceph-mon[102844]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:51 smithi196 ceph-mon[76349]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:42:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:42:52.059 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:42:52.060 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:42:52.060 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:42:52.060 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:42:52.060 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:42:52.372 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:42:52.372 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:52.372 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:52.372 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:52.372 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[102844]: from='client.47424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[102844]: from='client.47430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[102844]: from='client.57401 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2145316720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2900281735' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[100622]: from='client.47424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[100622]: from='client.47430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[100622]: from='client.57401 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2145316720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:52.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2900281735' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:52 smithi196 ceph-mon[76349]: from='client.47424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:52 smithi196 ceph-mon[76349]: from='client.47430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:52 smithi196 ceph-mon[76349]: from='client.57401 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2145316720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2900281735' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:53.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:53 smithi196 ceph-mon[76349]: from='client.57413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:53.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:53 smithi196 ceph-mon[76349]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-mon[100622]: from='client.57413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-mon[100622]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-mon[102844]: from='client.57413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-mon[102844]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:53.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:53.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:42:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:42:53.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:42:55.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:55 smithi184 ceph-mon[100622]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:55.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:55 smithi184 ceph-mon[102844]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:55 smithi196 ceph-mon[76349]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:57 smithi196 ceph-mon[76349]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:58.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:57 smithi184 ceph-mon[100622]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:58.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:57 smithi184 ceph-mon[102844]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:58.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:58.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:59 smithi196 ceph-mon[76349]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:59 smithi184 ceph-mon[100622]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:59 smithi184 ceph-mon[102844]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:00.292Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:00.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:00.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:01 smithi196 ceph-mon[76349]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:01 smithi184 ceph-mon[100622]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:01 smithi184 ceph-mon[102844]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:43:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[102844]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[100622]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:04.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:03.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:03.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:03.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:03 smithi196 ceph-mon[76349]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:05 smithi184 ceph-mon[100622]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:05 smithi184 ceph-mon[102844]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:05 smithi196 ceph-mon[76349]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:07 smithi184 ceph-mon[100622]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:07 smithi184 ceph-mon[102844]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:07 smithi196 ceph-mon[76349]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:09 smithi184 ceph-mon[100622]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:09 smithi184 ceph-mon[102844]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:09 smithi196 ceph-mon[76349]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:10.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:10.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:10.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:11 smithi196 ceph-mon[76349]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:11 smithi184 ceph-mon[100622]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:11 smithi184 ceph-mon[102844]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:43:14.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:13 smithi196 ceph-mon[76349]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:14.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-mon[100622]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-mon[102844]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:13.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:13.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:13.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:15 smithi184 ceph-mon[102844]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:15 smithi184 ceph-mon[100622]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:15 smithi196 ceph-mon[76349]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:17 smithi184 ceph-mon[100622]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:17 smithi184 ceph-mon[102844]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:17 smithi196 ceph-mon[76349]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:19 smithi184 ceph-mon[100622]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:19 smithi184 ceph-mon[102844]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:19 smithi196 ceph-mon[76349]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:20.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:20.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:20.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:21 smithi184 ceph-mon[100622]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:21 smithi184 ceph-mon[102844]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:21 smithi196 ceph-mon[76349]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:43:22.662 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:43:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[102844]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[102844]: from='client.47460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[100622]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:22 smithi184 ceph-mon[100622]: from='client.47460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:22 smithi196 ceph-mon[76349]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:22 smithi196 ceph-mon[76349]: from='client.47460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:23.197 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 60m - - 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 59m - - 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (54m) 1s ago 55m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (55m) 19s ago 55m 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (48m) 1s ago 53m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (48m) 19s ago 59m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (49m) 1s ago 60m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (47m) 1s ago 61m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (46m) 19s ago 59m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (47m) 1s ago 59m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:43:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (54m) 1s ago 54m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (54m) 19s ago 54m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (45m) 1s ago 58m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (44m) 1s ago 58m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (44m) 1s ago 58m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (43m) 1s ago 57m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (42m) 19s ago 57m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (41m) 19s ago 56m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (40m) 19s ago 56m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (39m) 19s ago 56m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (48m) 19s ago 55m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (39m) 1s ago 53m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (39m) 19s ago 53m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (39m) 1s ago 53m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:43:23.199 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (39m) 19s ago 53m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:43:23.511 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:43:23.512 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:43:23.782 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:43:23.783 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:43:23.783 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:43:23.783 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:23.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:23.783 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:23.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:23.783 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:23.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:24.097 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:43:24.097 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:24.097 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:24.098 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:24.098 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='client.57431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='client.57437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1068535367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/269316446' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='client.57431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='client.57437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1068535367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/269316446' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='client.57431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='client.57437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1068535367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:24.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/269316446' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:25 smithi196 ceph-mon[76349]: from='client.57449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:25.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:25 smithi196 ceph-mon[76349]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:25 smithi184 ceph-mon[100622]: from='client.57449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:25 smithi184 ceph-mon[100622]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:25 smithi184 ceph-mon[102844]: from='client.57449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:25 smithi184 ceph-mon[102844]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:27.675 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:27 smithi184 ceph-mon[100622]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:27.676 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:27 smithi184 ceph-mon[102844]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:27 smithi196 ceph-mon[76349]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:29 smithi184 ceph-mon[100622]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:29 smithi184 ceph-mon[102844]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:29 smithi196 ceph-mon[76349]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:30.294Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:30.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:30.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:31 smithi184 ceph-mon[100622]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:31 smithi184 ceph-mon[102844]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:31 smithi196 ceph-mon[76349]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:43:33.679 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:33 smithi196 ceph-mon[76349]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:33.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:33 smithi184 ceph-mon[100622]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:33.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:33 smithi184 ceph-mon[102844]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:33.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:33.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:33.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:35 smithi184 ceph-mon[100622]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:35 smithi184 ceph-mon[102844]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:35 smithi196 ceph-mon[76349]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:37 smithi184 ceph-mon[100622]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:37 smithi184 ceph-mon[102844]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:37 smithi196 ceph-mon[76349]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:38.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:38.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:39 smithi184 ceph-mon[102844]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:39 smithi184 ceph-mon[100622]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:39 smithi196 ceph-mon[76349]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:40.294Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:40.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:40.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:41 smithi184 ceph-mon[102844]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:41 smithi184 ceph-mon[100622]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:41 smithi196 ceph-mon[76349]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:43:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[100622]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:42 smithi184 ceph-mon[102844]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:42 smithi196 ceph-mon[76349]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:43.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:43.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:43.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:43:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:45 smithi184 ceph-mon[100622]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:45 smithi184 ceph-mon[102844]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:45 smithi196 ceph-mon[76349]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:47 smithi184 ceph-mon[100622]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:47 smithi184 ceph-mon[102844]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:47 smithi196 ceph-mon[76349]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:49 smithi184 ceph-mon[102844]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:49 smithi184 ceph-mon[100622]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:49 smithi196 ceph-mon[76349]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:50.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:50.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:50.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:51 smithi184 ceph-mon[102844]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:51 smithi184 ceph-mon[100622]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:51 smithi196 ceph-mon[76349]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:43:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:43:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:53 smithi184 ceph-mon[102844]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:53 smithi184 ceph-mon[100622]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:53.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:53.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:43:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:43:53.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:43:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:53 smithi196 ceph-mon[76349]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:54.398 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 60m - - 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 60m - - 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (54m) 12s ago 55m 29.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (55m) 11s ago 55m 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (48m) 12s ago 53m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (48m) 11s ago 59m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (49m) 12s ago 61m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (47m) 12s ago 61m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (47m) 11s ago 60m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (47m) 12s ago 60m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:43:54.923 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (55m) 12s ago 55m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (55m) 11s ago 55m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (46m) 12s ago 59m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (45m) 12s ago 59m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (44m) 12s ago 58m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (44m) 12s ago 58m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (42m) 11s ago 57m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (42m) 11s ago 57m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (40m) 11s ago 57m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (40m) 11s ago 56m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (49m) 11s ago 56m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (39m) 12s ago 54m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (39m) 11s ago 54m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (39m) 12s ago 54m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:43:54.924 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (39m) 11s ago 53m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:55.232 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:43:55.233 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:43:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:43:55.505 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:43:55.505 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:43:55.505 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:43:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[100622]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[100622]: from='client.57461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[100622]: from='client.57467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2477956352' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[102844]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[102844]: from='client.57461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[102844]: from='client.57467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2477956352' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:55.814 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:43:55.814 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:55.814 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:55.814 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:55.814 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:55 smithi196 ceph-mon[76349]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:55 smithi196 ceph-mon[76349]: from='client.57461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:55 smithi196 ceph-mon[76349]: from='client.57467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2477956352' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:56.864 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[102844]: from='client.57473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.864 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[102844]: from='client.57485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.864 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3905621445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:56.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[100622]: from='client.57473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[100622]: from='client.57485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3905621445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:56 smithi196 ceph-mon[76349]: from='client.57473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:56 smithi196 ceph-mon[76349]: from='client.57485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3905621445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:57 smithi184 ceph-mon[100622]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:57 smithi184 ceph-mon[102844]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:57 smithi196 ceph-mon[76349]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:58.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:58.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:59 smithi184 ceph-mon[102844]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:59 smithi184 ceph-mon[100622]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:59 smithi196 ceph-mon[76349]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:00.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:00.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:00.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:01 smithi196 ceph-mon[76349]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:01 smithi184 ceph-mon[100622]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:01 smithi184 ceph-mon[102844]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:02.258 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:44:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:03 smithi196 ceph-mon[76349]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[100622]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-mon[102844]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:03.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:03.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:03.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:05 smithi184 ceph-mon[100622]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:44:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:05 smithi184 ceph-mon[102844]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:44:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:05 smithi196 ceph-mon[76349]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T20:44:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:07 smithi184 ceph-mon[100622]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:44:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:07 smithi184 ceph-mon[102844]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:44:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:07 smithi196 ceph-mon[76349]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:44:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:09 smithi184 ceph-mon[100622]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:44:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:09 smithi184 ceph-mon[102844]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:44:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:09 smithi196 ceph-mon[76349]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-20T20:44:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:10.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:10.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:10.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:11 smithi184 ceph-mon[100622]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:44:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:11 smithi184 ceph-mon[102844]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:44:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:11 smithi196 ceph-mon[76349]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:44:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:44:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:13 smithi184 ceph-mon[100622]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:44:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:13 smithi184 ceph-mon[102844]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:44:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:13.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:13.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:13.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:14.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:13 smithi196 ceph-mon[76349]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T20:44:14.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:15.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:15 smithi196 ceph-mon[76349]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:15 smithi184 ceph-mon[100622]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:15 smithi184 ceph-mon[102844]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:17.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:17 smithi184 ceph-mon[100622]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:44:17.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:17 smithi184 ceph-mon[102844]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:44:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:17 smithi196 ceph-mon[76349]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:44:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:20.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:19 smithi184 ceph-mon[100622]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:20.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:19 smithi184 ceph-mon[102844]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:19 smithi196 ceph-mon[76349]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T20:44:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:20.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:20.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:20.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:21 smithi184 ceph-mon[100622]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:44:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:21 smithi184 ceph-mon[102844]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:44:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:21 smithi196 ceph-mon[76349]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T20:44:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[100622]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-mon[102844]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:23 smithi196 ceph-mon[76349]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:23.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:23.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:23.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:25.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:24 smithi196 ceph-mon[76349]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:44:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[100622]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:44:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:24 smithi184 ceph-mon[102844]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:44:26.112 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 61m - - 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 60m - - 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (55m) 4s ago 56m 29.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (56m) 2s ago 56m 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:44:26.642 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (49m) 4s ago 54m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (49m) 2s ago 60m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (50m) 4s ago 62m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (48m) 4s ago 62m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (47m) 2s ago 60m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (48m) 4s ago 60m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (56m) 4s ago 55m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (55m) 2s ago 55m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (46m) 4s ago 59m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (45m) 4s ago 59m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (45m) 4s ago 59m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (44m) 4s ago 58m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (43m) 2s ago 58m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (42m) 2s ago 58m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (41m) 2s ago 57m 65.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:44:26.643 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (40m) 2s ago 57m 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:44:26.644 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (49m) 2s ago 56m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:44:26.644 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (40m) 4s ago 54m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:44:26.644 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (40m) 2s ago 55m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:44:26.644 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (40m) 4s ago 54m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:44:26.644 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (40m) 2s ago 54m 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:44:26.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:44:26.952 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:44:27.220 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:44:27.221 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:44:27.533 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:44:27.533 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:27.533 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:27.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[100622]: from='client.57497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[100622]: from='client.57503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[100622]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[100622]: from='client.57509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3663790143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[102844]: from='client.57497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[102844]: from='client.57503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[102844]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[102844]: from='client.57509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3663790143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:27 smithi196 ceph-mon[76349]: from='client.57497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:27 smithi196 ceph-mon[76349]: from='client.57503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:27 smithi196 ceph-mon[76349]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:44:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:27 smithi196 ceph-mon[76349]: from='client.57509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3663790143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[102844]: from='client.47556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2242665171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[100622]: from='client.47556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2242665171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:28 smithi196 ceph-mon[76349]: from='client.47556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2242665171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:29 smithi184 ceph-mon[102844]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:44:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:29 smithi184 ceph-mon[100622]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:44:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:29 smithi196 ceph-mon[76349]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:44:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:30.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:30.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:30.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:31 smithi184 ceph-mon[100622]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:31 smithi184 ceph-mon[102844]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:31 smithi196 ceph-mon[76349]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:44:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:44:33.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:33 smithi184 ceph-mon[100622]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:44:33.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:33 smithi184 ceph-mon[102844]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:44:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:33 smithi196 ceph-mon[76349]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:44:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:33.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:33.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:33.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:35 smithi196 ceph-mon[76349]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T20:44:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:35 smithi184 ceph-mon[100622]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T20:44:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:35 smithi184 ceph-mon[102844]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T20:44:37.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:37 smithi184 ceph-mon[100622]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T20:44:37.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:37 smithi184 ceph-mon[102844]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T20:44:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:37 smithi196 ceph-mon[76349]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T20:44:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:39 smithi184 ceph-mon[100622]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-20T20:44:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:39 smithi184 ceph-mon[102844]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-20T20:44:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:39 smithi196 ceph-mon[76349]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-20T20:44:40.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:40.298Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:40.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:40.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:40.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:40.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:41 smithi184 ceph-mon[100622]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:44:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:41 smithi184 ceph-mon[102844]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:44:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:41 smithi196 ceph-mon[76349]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-20T20:44:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:44:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:43 smithi196 ceph-mon[76349]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:43.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[100622]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:43.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-mon[102844]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:43.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:43.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:43.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:44:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:45 smithi196 ceph-mon[76349]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:45 smithi184 ceph-mon[100622]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:45 smithi184 ceph-mon[102844]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:47.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:47 smithi184 ceph-mon[100622]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:47.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:47 smithi184 ceph-mon[102844]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:47 smithi196 ceph-mon[76349]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:49 smithi184 ceph-mon[100622]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:49 smithi184 ceph-mon[102844]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:49 smithi196 ceph-mon[76349]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:50.299Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:50.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:50.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:51 smithi184 ceph-mon[100622]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:51 smithi184 ceph-mon[102844]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:51 smithi196 ceph-mon[76349]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:44:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:44:53.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:53 smithi184 ceph-mon[100622]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:53.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:53 smithi184 ceph-mon[102844]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:53 smithi196 ceph-mon[76349]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:53.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:53.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:44:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:44:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:44:53.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:44:55.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:55 smithi196 ceph-mon[76349]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:55 smithi184 ceph-mon[100622]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:55 smithi184 ceph-mon[102844]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:57.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:57 smithi184 ceph-mon[100622]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:57.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:57 smithi184 ceph-mon[102844]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:57.827 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:44:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:57 smithi196 ceph-mon[76349]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:58.350 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 61m - - 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 61m - - 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (55m) 16s ago 56m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (56m) 14s ago 56m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (49m) 16s ago 54m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (49m) 14s ago 60m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (50m) 16s ago 62m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (48m) 16s ago 62m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (48m) 14s ago 61m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (48m) 16s ago 61m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:44:58.351 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (56m) 16s ago 56m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (56m) 14s ago 56m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (47m) 16s ago 60m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (46m) 16s ago 60m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (45m) 16s ago 59m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (45m) 16s ago 59m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (43m) 14s ago 58m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (43m) 14s ago 58m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (41m) 14s ago 58m 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (41m) 14s ago 57m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (50m) 14s ago 57m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (40m) 16s ago 55m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (40m) 14s ago 55m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (40m) 16s ago 55m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:44:58.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (40m) 14s ago 55m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:44:58.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:58.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:44:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:44:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:44:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:44:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:44:58.675 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:44:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:58.953 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:44:58.953 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:44:58.953 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:44:58.953 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:44:58.954 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:44:59.267 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:44:59.267 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:59.267 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:59.267 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:59.267 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='client.57533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='client.47574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='client.57545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/650509228' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/332504276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='client.57533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='client.47574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='client.57545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/650509228' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:44:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/332504276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='client.57533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='client.47574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='client.57545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/650509228' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:44:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/332504276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:00.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:00.300Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:00.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:00.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:00.477 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:00.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:00 smithi184 ceph-mon[100622]: from='client.47592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:00 smithi184 ceph-mon[102844]: from='client.47592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:00 smithi196 ceph-mon[76349]: from='client.47592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:01 smithi184 ceph-mon[102844]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:01 smithi184 ceph-mon[100622]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:01 smithi196 ceph-mon[76349]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:45:03.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[102844]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:03.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.732 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[100622]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:03.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:03 smithi196 ceph-mon[76349]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:04.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:03.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:03.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:03.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[102844]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:05 smithi184 ceph-mon[100622]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:05 smithi196 ceph-mon[76349]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:07.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:07 smithi184 ceph-mon[100622]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:07.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:07 smithi184 ceph-mon[102844]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:07 smithi196 ceph-mon[76349]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:09 smithi184 ceph-mon[100622]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:09 smithi184 ceph-mon[102844]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:09 smithi196 ceph-mon[76349]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:10.300Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:10.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:10.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:11 smithi184 ceph-mon[100622]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:11 smithi184 ceph-mon[102844]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:11 smithi196 ceph-mon[76349]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:45:13.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:13 smithi184 ceph-mon[100622]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:13.733 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:13 smithi184 ceph-mon[102844]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:13 smithi196 ceph-mon[76349]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:13.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:13.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:13.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:15.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:15 smithi196 ceph-mon[76349]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:15 smithi184 ceph-mon[100622]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:15 smithi184 ceph-mon[102844]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:17.680 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:17 smithi196 ceph-mon[76349]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:17.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:17 smithi184 ceph-mon[100622]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:17.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:17 smithi184 ceph-mon[102844]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:19.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:19 smithi184 ceph-mon[100622]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:19.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:19 smithi184 ceph-mon[102844]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:19 smithi196 ceph-mon[76349]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:20.396 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:20.301Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:20.396 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:20.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:20.396 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:20.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:21.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:21 smithi184 ceph-mon[100622]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:21.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:21 smithi184 ceph-mon[102844]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:21 smithi196 ceph-mon[76349]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:45:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:23 smithi196 ceph-mon[76349]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:23.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[100622]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[102844]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:23.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:23.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:23.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:25 smithi196 ceph-mon[76349]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[100622]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:25 smithi184 ceph-mon[102844]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:27.727 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:27 smithi196 ceph-mon[76349]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:27.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:27 smithi184 ceph-mon[100622]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:27.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:27 smithi184 ceph-mon[102844]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:29.570 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:45:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:29 smithi184 ceph-mon[100622]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:29 smithi184 ceph-mon[102844]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:29 smithi196 ceph-mon[76349]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:30.100 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:45:30.100 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 62m - - 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 61m - - 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (56m) 7s ago 57m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (57m) 5s ago 57m 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (50m) 7s ago 55m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (50m) 5s ago 61m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (51m) 7s ago 63m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (49m) 7s ago 63m 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (48m) 5s ago 61m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (49m) 7s ago 61m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (57m) 7s ago 57m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (56m) 5s ago 56m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (47m) 7s ago 61m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:45:30.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (46m) 7s ago 60m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (46m) 7s ago 60m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (45m) 7s ago 59m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (44m) 5s ago 59m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (43m) 5s ago 59m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (42m) 5s ago 58m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (41m) 5s ago 58m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (50m) 5s ago 57m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (41m) 7s ago 55m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (41m) 5s ago 56m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (41m) 7s ago 55m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:45:30.102 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (41m) 5s ago 55m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:45:30.411 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:45:30.412 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:45:30.412 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:30.302Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:30.412 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:30.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:30.412 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:30.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:45:30.680 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:45:30.681 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:45:30.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-mon[100622]: from='client.57569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:30.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1422861669' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:30.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-mon[102844]: from='client.57569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:30.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1422861669' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:30 smithi196 ceph-mon[76349]: from='client.57569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1422861669' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:30.989 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:45:30.990 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:30.990 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:30.990 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:30.990 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[100622]: from='client.47610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[100622]: from='client.57581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[100622]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[100622]: from='client.47628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3904657248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[102844]: from='client.47610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[102844]: from='client.57581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[102844]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[102844]: from='client.47628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3904657248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:31 smithi196 ceph-mon[76349]: from='client.47610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:31 smithi196 ceph-mon[76349]: from='client.57581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:31 smithi196 ceph-mon[76349]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:31 smithi196 ceph-mon[76349]: from='client.47628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3904657248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:45:33.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:33 smithi184 ceph-mon[100622]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:33.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:33 smithi184 ceph-mon[102844]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:33 smithi196 ceph-mon[76349]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:33.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:33.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:33.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:35 smithi184 ceph-mon[100622]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:35 smithi184 ceph-mon[102844]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:35.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:35 smithi196 ceph-mon[76349]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:37.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:37 smithi196 ceph-mon[76349]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:37.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:37 smithi184 ceph-mon[100622]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:37.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:37 smithi184 ceph-mon[102844]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:38.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:39 smithi196 ceph-mon[76349]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:39 smithi184 ceph-mon[100622]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:39 smithi184 ceph-mon[102844]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:40.303Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:40.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:40.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:41 smithi184 ceph-mon[100622]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:41 smithi184 ceph-mon[102844]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:41 smithi196 ceph-mon[76349]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:42.116 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[102844]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[100622]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:43.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:43.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:43.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:43 smithi196 ceph-mon[76349]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:44.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:45 smithi196 ceph-mon[76349]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[100622]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:45:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:45 smithi184 ceph-mon[102844]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:47.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:47 smithi184 ceph-mon[100622]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:47.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:47 smithi184 ceph-mon[102844]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:47 smithi196 ceph-mon[76349]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:49 smithi184 ceph-mon[100622]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:49 smithi184 ceph-mon[102844]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:49 smithi196 ceph-mon[76349]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:50.304Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:50.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:50.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:51 smithi184 ceph-mon[100622]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:51 smithi184 ceph-mon[102844]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:51 smithi196 ceph-mon[76349]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:45:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:45:53.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:53 smithi184 ceph-mon[100622]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:53.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:53 smithi184 ceph-mon[102844]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:53 smithi196 ceph-mon[76349]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:53.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:53.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:45:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:45:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:45:53.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:45:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:55 smithi184 ceph-mon[100622]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:55 smithi184 ceph-mon[102844]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:55 smithi196 ceph-mon[76349]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:57.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:57 smithi196 ceph-mon[76349]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:57.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:57 smithi184 ceph-mon[100622]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:57.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:57 smithi184 ceph-mon[102844]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[100622]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[102844]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:59 smithi196 ceph-mon[76349]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:45:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:00.663 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:00.304Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:00.663 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:00.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:00.663 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:00.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:01.293 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:46:01.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:01 smithi184 ceph-mon[100622]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:01.741 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:01 smithi184 ceph-mon[102844]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 62m - - 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 62m - - 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (56m) 19s ago 58m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (57m) 17s ago 57m 91.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (50m) 19s ago 56m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (50m) 17s ago 61m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (51m) 19s ago 63m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (50m) 19s ago 63m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (49m) 17s ago 62m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:46:01.844 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (49m) 19s ago 62m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (57m) 19s ago 57m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (57m) 17s ago 57m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (48m) 19s ago 61m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (47m) 19s ago 61m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (46m) 19s ago 60m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (46m) 19s ago 60m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (44m) 17s ago 60m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (44m) 17s ago 59m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (42m) 17s ago 59m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (42m) 17s ago 58m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (51m) 17s ago 58m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (41m) 19s ago 56m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (41m) 17s ago 56m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (41m) 19s ago 56m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:46:01.845 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (41m) 17s ago 56m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:46:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:01 smithi196 ceph-mon[76349]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:02.206 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:46:02.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:46:02.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:46:02.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:02.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:46:02.207 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:46:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:46:02.508 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:46:02.509 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:46:02.509 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:46:02.509 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:46:02.509 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:46:02.509 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:46:02.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[100622]: from='client.57605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[100622]: from='client.57611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3363421687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:02.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[102844]: from='client.57605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[102844]: from='client.57611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3363421687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:02.858 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:46:02.858 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:02.858 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:02.858 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:02.859 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:02 smithi196 ceph-mon[76349]: from='client.57605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:02 smithi196 ceph-mon[76349]: from='client.57611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3363421687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='client.57617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='client.47664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/177878310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:03.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:03.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:03.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:03.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='client.57617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='client.47664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/177878310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='client.57617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='client.47664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/177878310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:03.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[100622]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[102844]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:05 smithi196 ceph-mon[76349]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:07.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:07 smithi184 ceph-mon[100622]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:07.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:07 smithi184 ceph-mon[102844]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:07 smithi196 ceph-mon[76349]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:09 smithi184 ceph-mon[102844]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:09 smithi184 ceph-mon[100622]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:09 smithi196 ceph-mon[76349]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:10.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:10.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:10.802 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:10.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:10.802 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:10.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:11 smithi184 ceph-mon[100622]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:11 smithi184 ceph-mon[102844]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:11 smithi196 ceph-mon[76349]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:46:13.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:13 smithi184 ceph-mon[100622]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:13.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:13 smithi184 ceph-mon[102844]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:13 smithi196 ceph-mon[76349]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:14.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:13.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:13.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:13.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:14.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:15 smithi184 ceph-mon[100622]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:15 smithi184 ceph-mon[102844]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:15 smithi196 ceph-mon[76349]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:17.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:17 smithi196 ceph-mon[76349]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:17 smithi184 ceph-mon[100622]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:17 smithi184 ceph-mon[102844]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:19.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:19.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:19 smithi196 ceph-mon[76349]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:19 smithi184 ceph-mon[100622]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:19 smithi184 ceph-mon[102844]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:20.306Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:20.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:20.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:21 smithi184 ceph-mon[100622]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:21 smithi184 ceph-mon[102844]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:21 smithi196 ceph-mon[76349]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:22.237 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:46:23.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[102844]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:23.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[100622]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:23.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:23 smithi196 ceph-mon[76349]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:23.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:23.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:23.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:25 smithi196 ceph-mon[76349]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:25.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:25.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[100622]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[102844]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:27 smithi196 ceph-mon[76349]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:27 smithi184 ceph-mon[100622]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:27 smithi184 ceph-mon[102844]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[100622]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[102844]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:29 smithi196 ceph-mon[76349]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:30.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:30.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:30.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:31 smithi184 ceph-mon[100622]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:31 smithi184 ceph-mon[102844]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:31 smithi196 ceph-mon[76349]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:46:33.158 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 63m - - 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 63m - - 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (57m) 10s ago 58m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (58m) 9s ago 58m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (51m) 10s ago 56m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (51m) 9s ago 62m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (52m) 10s ago 64m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (50m) 10s ago 64m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (49m) 9s ago 62m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (50m) 10s ago 62m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (58m) 10s ago 58m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (58m) 9s ago 58m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:46:33.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (48m) 10s ago 62m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (47m) 10s ago 61m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (47m) 10s ago 61m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (46m) 10s ago 60m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (45m) 9s ago 60m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (44m) 9s ago 60m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (43m) 9s ago 59m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (42m) 9s ago 59m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (51m) 9s ago 58m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (42m) 10s ago 57m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (42m) 9s ago 57m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (42m) 10s ago 56m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:46:33.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (42m) 9s ago 56m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:46:33.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:33 smithi184 ceph-mon[102844]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:33.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:33 smithi184 ceph-mon[100622]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:33 smithi196 ceph-mon[76349]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:33.998 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:46:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:46:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:46:34.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:46:34.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:46:34.001 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:46:34.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:33.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:34.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:33.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:34.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:33.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:34.266 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:46:34.266 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:46:34.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[100622]: from='client.57641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[100622]: from='client.57647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[100622]: from='client.57653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3168142831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[102844]: from='client.57641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[102844]: from='client.57647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[102844]: from='client.57653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3168142831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:34.606 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:46:34.606 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:34.606 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:34.606 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:34.607 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:34 smithi196 ceph-mon[76349]: from='client.57641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:34 smithi196 ceph-mon[76349]: from='client.57647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:34 smithi196 ceph-mon[76349]: from='client.57653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3168142831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:35.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[100622]: from='client.57665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[100622]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:35.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1545760858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[102844]: from='client.57665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[102844]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:35.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1545760858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:35 smithi196 ceph-mon[76349]: from='client.57665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:35 smithi196 ceph-mon[76349]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1545760858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:37 smithi184 ceph-mon[100622]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:37 smithi184 ceph-mon[102844]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:37.844 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:37 smithi196 ceph-mon[76349]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:39 smithi184 ceph-mon[100622]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:39 smithi184 ceph-mon[102844]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:39 smithi196 ceph-mon[76349]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:40.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:40.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:40.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:41 smithi184 ceph-mon[100622]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:41 smithi184 ceph-mon[102844]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:41 smithi196 ceph-mon[76349]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:42.288 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:46:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[100622]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[102844]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:43 smithi196 ceph-mon[76349]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:43.907 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:43.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:43.907 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:43.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:43.907 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:43.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:44.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:44.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[102844]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[100622]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:45 smithi196 ceph-mon[76349]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:46:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:47 smithi184 ceph-mon[100622]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:47 smithi184 ceph-mon[102844]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:47 smithi196 ceph-mon[76349]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:49 smithi184 ceph-mon[100622]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:49 smithi184 ceph-mon[102844]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:49 smithi196 ceph-mon[76349]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:50.308Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:50.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:50.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:51 smithi184 ceph-mon[100622]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:51 smithi184 ceph-mon[102844]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:51 smithi196 ceph-mon[76349]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:46:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:46:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:53 smithi184 ceph-mon[100622]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:53 smithi184 ceph-mon[102844]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:53.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:53.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:46:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:46:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:46:53.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:46:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:53 smithi196 ceph-mon[76349]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:55 smithi184 ceph-mon[100622]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:55 smithi184 ceph-mon[102844]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:55 smithi196 ceph-mon[76349]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:57 smithi184 ceph-mon[100622]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:57 smithi184 ceph-mon[102844]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:57 smithi196 ceph-mon[76349]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:59.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:59.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:59 smithi196 ceph-mon[76349]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:59.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[100622]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[102844]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:00.309Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:00.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:00.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:01 smithi196 ceph-mon[76349]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:01 smithi184 ceph-mon[100622]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:01 smithi184 ceph-mon[102844]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[100622]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[102844]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:03.993 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:03.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:03.993 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:03.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:03.993 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:03.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:03 smithi196 ceph-mon[76349]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:04.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 63m - - 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 63m - - 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (57m) 2s ago 59m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (58m) 0s ago 58m 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (52m) 2s ago 57m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (51m) 0s ago 62m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (52m) 2s ago 64m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:47:05.443 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (51m) 2s ago 64m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (50m) 0s ago 63m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (50m) 2s ago 63m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (58m) 2s ago 58m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (58m) 0s ago 58m 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (49m) 2s ago 62m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (48m) 2s ago 62m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (47m) 2s ago 61m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (47m) 2s ago 61m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (46m) 0s ago 61m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (45m) 0s ago 60m 62.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (44m) 0s ago 60m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (43m) 0s ago 59m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (52m) 0s ago 59m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (43m) 2s ago 57m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:47:05.444 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (42m) 0s ago 57m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:47:05.445 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (42m) 2s ago 57m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:47:05.445 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (42m) 0s ago 57m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:47:05.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:47:05.753 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:47:06.022 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:47:06.023 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:47:06.023 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:47:06.023 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:47:06.023 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:47:06.023 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:47:06.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[100622]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:06.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[102844]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:06.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:05 smithi196 ceph-mon[76349]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:06.331 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:47:06.331 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:06.331 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:06.331 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:06.331 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[100622]: from='client.57677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[100622]: from='client.57683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[100622]: from='client.57689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/331761769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/65571031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[102844]: from='client.57677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[102844]: from='client.57683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[102844]: from='client.57689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/331761769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:07.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/65571031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:06 smithi196 ceph-mon[76349]: from='client.57677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:06 smithi196 ceph-mon[76349]: from='client.57683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:06 smithi196 ceph-mon[76349]: from='client.57689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/331761769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/65571031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:07 smithi184 ceph-mon[100622]: from='client.47736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:07 smithi184 ceph-mon[100622]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:07 smithi184 ceph-mon[102844]: from='client.47736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:07 smithi184 ceph-mon[102844]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:07 smithi196 ceph-mon[76349]: from='client.47736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:07 smithi196 ceph-mon[76349]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:09 smithi184 ceph-mon[100622]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:09 smithi184 ceph-mon[102844]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:09 smithi196 ceph-mon[76349]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:10.310Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:10.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:10.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:11 smithi184 ceph-mon[100622]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:11 smithi184 ceph-mon[102844]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:11 smithi196 ceph-mon[76349]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:47:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-mon[100622]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-mon[102844]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:13.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:13.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:13.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:13 smithi196 ceph-mon[76349]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:15 smithi184 ceph-mon[100622]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:15 smithi184 ceph-mon[102844]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:15 smithi196 ceph-mon[76349]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:17 smithi184 ceph-mon[100622]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:17 smithi184 ceph-mon[102844]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:17 smithi196 ceph-mon[76349]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:19.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:19 smithi196 ceph-mon[76349]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:19 smithi184 ceph-mon[100622]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:19 smithi184 ceph-mon[102844]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:20.310Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:20.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:20.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:21 smithi184 ceph-mon[100622]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:21 smithi184 ceph-mon[102844]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:21 smithi196 ceph-mon[76349]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:47:24.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[100622]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:24.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[102844]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:24.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.039 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:23.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:24.039 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:23.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:24.039 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:23.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:23 smithi196 ceph-mon[76349]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[100622]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[102844]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:25 smithi196 ceph-mon[76349]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:26.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:26.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:27 smithi184 ceph-mon[100622]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:27 smithi184 ceph-mon[102844]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:27 smithi196 ceph-mon[76349]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:29 smithi184 ceph-mon[102844]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:29 smithi184 ceph-mon[100622]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:29 smithi196 ceph-mon[76349]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:30.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:30.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:30.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:31 smithi184 ceph-mon[100622]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:31 smithi184 ceph-mon[102844]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:47:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:31 smithi196 ceph-mon[76349]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:33 smithi184 ceph-mon[100622]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:33 smithi184 ceph-mon[102844]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:33.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:33.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:33.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:33 smithi196 ceph-mon[76349]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:35 smithi184 ceph-mon[100622]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:36.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:35 smithi184 ceph-mon[102844]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:35 smithi196 ceph-mon[76349]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:36.628 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 64m - - 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 64m - - 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (58m) 13s ago 59m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (59m) 12s ago 59m 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (52m) 13s ago 57m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (52m) 12s ago 63m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (53m) 13s ago 65m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (51m) 13s ago 65m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (51m) 12s ago 63m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:47:37.156 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (51m) 13s ago 63m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (59m) 13s ago 59m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (59m) 12s ago 59m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (49m) 13s ago 63m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (49m) 13s ago 62m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (48m) 13s ago 62m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (47m) 13s ago 61m 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (46m) 12s ago 61m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (45m) 12s ago 61m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (44m) 12s ago 60m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (43m) 12s ago 60m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (52m) 12s ago 59m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (43m) 13s ago 58m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (43m) 12s ago 58m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (43m) 13s ago 57m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:47:37.157 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (43m) 12s ago 57m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:47:37.473 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:47:37.474 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:47:37.743 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:47:37.744 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:47:37.744 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:47:37.744 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:47:37.744 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:47:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[102844]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[102844]: from='client.47748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2454308642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[100622]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[100622]: from='client.47748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2454308642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:38.053 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:47:38.053 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:38.053 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:38.053 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:38.053 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:37 smithi196 ceph-mon[76349]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:37 smithi196 ceph-mon[76349]: from='client.47748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2454308642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[100622]: from='client.57719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[100622]: from='client.57725 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[100622]: from='client.57737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3183289861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[102844]: from='client.57719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[102844]: from='client.57725 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[102844]: from='client.57737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3183289861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:38 smithi196 ceph-mon[76349]: from='client.57719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:38 smithi196 ceph-mon[76349]: from='client.57725 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:38 smithi196 ceph-mon[76349]: from='client.57737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:39.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3183289861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:40.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:40.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:39 smithi196 ceph-mon[76349]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:39 smithi184 ceph-mon[100622]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:39 smithi184 ceph-mon[102844]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:40.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:40.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:40.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:41 smithi196 ceph-mon[76349]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:41 smithi184 ceph-mon[102844]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:41 smithi184 ceph-mon[100622]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:47:43.921 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:43.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:43.922 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:43.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:43.922 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:43.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[100622]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:43.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[102844]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:43.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:43.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:43 smithi196 ceph-mon[76349]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[100622]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[102844]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:46.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:45 smithi196 ceph-mon[76349]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:47:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:47 smithi184 ceph-mon[100622]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:47 smithi184 ceph-mon[102844]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:47 smithi196 ceph-mon[76349]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:49 smithi184 ceph-mon[102844]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:49 smithi184 ceph-mon[100622]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:49 smithi196 ceph-mon[76349]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:50.312Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:50.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:50.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:51 smithi196 ceph-mon[76349]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:51 smithi184 ceph-mon[100622]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:51 smithi184 ceph-mon[102844]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:47:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:47:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:52 smithi196 ceph-mon[76349]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:52 smithi184 ceph-mon[100622]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:52 smithi184 ceph-mon[102844]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:54.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:53.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:53.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:47:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:47:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:47:53.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:47:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:55 smithi184 ceph-mon[100622]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:55 smithi184 ceph-mon[102844]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:55 smithi196 ceph-mon[76349]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:57 smithi184 ceph-mon[100622]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:57 smithi184 ceph-mon[102844]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:57 smithi196 ceph-mon[76349]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:59.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[100622]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:59.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:59.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[102844]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:59 smithi196 ceph-mon[76349]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:00.312Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:00.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:00.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:01 smithi184 ceph-mon[100622]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:01 smithi184 ceph-mon[102844]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:01.843 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:01 smithi196 ceph-mon[76349]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:48:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:03 smithi184 ceph-mon[100622]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:03 smithi184 ceph-mon[102844]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:03.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:03.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:03.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:03.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:03 smithi196 ceph-mon[76349]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[100622]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[102844]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:05.841 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:05.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:05 smithi196 ceph-mon[76349]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:06.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:06.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:07 smithi184 ceph-mon[100622]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:07 smithi184 ceph-mon[102844]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:07 smithi196 ceph-mon[76349]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:08.350 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 64m - - 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 64m - - 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (59m) 5s ago 60m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (59m) 3s ago 59m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (53m) 5s ago 58m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (52m) 3s ago 63m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (53m) 5s ago 65m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:48:08.882 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (52m) 5s ago 65m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (51m) 3s ago 64m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (51m) 5s ago 64m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (59m) 5s ago 59m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (59m) 3s ago 59m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (50m) 5s ago 63m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (49m) 5s ago 63m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (48m) 5s ago 62m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (48m) 5s ago 62m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (47m) 3s ago 62m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (46m) 3s ago 61m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (45m) 3s ago 61m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (44m) 3s ago 61m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (53m) 3s ago 60m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (44m) 5s ago 58m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:48:08.883 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (43m) 3s ago 58m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:48:08.884 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (43m) 5s ago 58m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:48:08.884 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (43m) 3s ago 58m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:48:09.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:48:09.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:48:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:48:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:48:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:48:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:48:09.463 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:48:09.775 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:48:09.775 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:09.775 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:09.775 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:09.775 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[100622]: from='client.47784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[100622]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[100622]: from='client.47790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/103242673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[102844]: from='client.47784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[102844]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[102844]: from='client.47790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/103242673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:09 smithi196 ceph-mon[76349]: from='client.47784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:09 smithi196 ceph-mon[76349]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:09 smithi196 ceph-mon[76349]: from='client.47790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/103242673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:10.720 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:10.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:10.720 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:10.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:10.720 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:10.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[100622]: from='client.47796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[100622]: from='client.57773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/419555159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[102844]: from='client.47796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[102844]: from='client.57773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/419555159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:10 smithi196 ceph-mon[76349]: from='client.47796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:10 smithi196 ceph-mon[76349]: from='client.57773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/419555159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:11 smithi184 ceph-mon[100622]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:11 smithi184 ceph-mon[102844]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:11 smithi196 ceph-mon[76349]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:48:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:13 smithi184 ceph-mon[100622]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:13 smithi184 ceph-mon[102844]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:13.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:13.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:13.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:13 smithi196 ceph-mon[76349]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:14.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:48:13.813432853Z level=info msg="Completed cleanup jobs" duration=13.278701ms 2024-09-20T20:48:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:15 smithi184 ceph-mon[100622]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:15 smithi184 ceph-mon[102844]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:15 smithi196 ceph-mon[76349]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:17 smithi184 ceph-mon[100622]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:17 smithi184 ceph-mon[102844]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:17 smithi196 ceph-mon[76349]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:19 smithi184 ceph-mon[100622]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:19 smithi184 ceph-mon[102844]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:19 smithi196 ceph-mon[76349]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:20.314Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:20.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:20.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:21 smithi184 ceph-mon[102844]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:21 smithi184 ceph-mon[100622]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:21 smithi196 ceph-mon[76349]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T20:48:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:23 smithi184 ceph-mon[102844]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:23 smithi184 ceph-mon[100622]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:23.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:23.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:23.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:23 smithi196 ceph-mon[76349]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:25.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:24 smithi196 ceph-mon[76349]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:25.173 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.173 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.173 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.173 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[100622]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:25.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:25.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:24 smithi184 ceph-mon[102844]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:26.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:27 smithi184 ceph-mon[100622]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:27 smithi184 ceph-mon[102844]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:27 smithi196 ceph-mon[76349]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[100622]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[102844]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:29 smithi196 ceph-mon[76349]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:30.315Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:30.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:30.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:31 smithi184 ceph-mon[100622]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:31 smithi184 ceph-mon[102844]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:31 smithi196 ceph-mon[76349]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:48:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:33 smithi184 ceph-mon[100622]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:33 smithi184 ceph-mon[102844]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:33.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:33.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:33.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:33.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:33 smithi196 ceph-mon[76349]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:35 smithi184 ceph-mon[100622]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:35 smithi184 ceph-mon[102844]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:35 smithi196 ceph-mon[76349]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:37 smithi184 ceph-mon[100622]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:37 smithi184 ceph-mon[102844]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:37 smithi196 ceph-mon[76349]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:39 smithi184 ceph-mon[102844]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:39 smithi184 ceph-mon[100622]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:39 smithi196 ceph-mon[76349]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:40.072 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:48:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:40.315Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:40.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:40.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 65m - - 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 65m - - 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (59m) 16s ago 60m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (60m) 15s ago 60m 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (53m) 16s ago 58m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (53m) 15s ago 64m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (54m) 16s ago 66m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (52m) 16s ago 66m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (52m) 15s ago 64m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:48:40.601 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (52m) 16s ago 64m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (60m) 16s ago 60m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (60m) 15s ago 60m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (50m) 16s ago 64m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (50m) 16s ago 63m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (49m) 16s ago 63m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (48m) 16s ago 63m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (47m) 15s ago 62m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (46m) 15s ago 62m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (45m) 15s ago 61m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (44m) 15s ago 61m 70.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (53m) 15s ago 61m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (44m) 16s ago 59m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (44m) 15s ago 59m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (44m) 16s ago 58m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:48:40.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (44m) 15s ago 58m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:48:40.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:48:40.912 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:48:41.186 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:48:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:48:41.187 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:48:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:48:41.496 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:48:41.496 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:41.497 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:41.497 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:41.497 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:41.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: from='client.47820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: from='client.47826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:41.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: from='client.57797 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/750877373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:41.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2648859569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: from='client.47820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: from='client.47826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: from='client.57797 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/750877373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2648859569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: from='client.47820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: from='client.47826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: from='client.57797 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/750877373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2648859569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:48:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:42 smithi196 ceph-mon[76349]: from='client.57809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:42 smithi184 ceph-mon[100622]: from='client.57809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:42 smithi184 ceph-mon[102844]: from='client.57809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:43 smithi196 ceph-mon[76349]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:43 smithi184 ceph-mon[100622]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:43 smithi184 ceph-mon[102844]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:43.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:43.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:43.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:43.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:43.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[100622]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[102844]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:45 smithi196 ceph-mon[76349]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:48:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:47 smithi196 ceph-mon[76349]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:47.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:47 smithi184 ceph-mon[100622]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:47.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:47 smithi184 ceph-mon[102844]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:49 smithi184 ceph-mon[100622]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:49 smithi184 ceph-mon[102844]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:49 smithi196 ceph-mon[76349]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:50.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:50.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:50.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:51 smithi184 ceph-mon[100622]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:51 smithi184 ceph-mon[102844]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:51 smithi196 ceph-mon[76349]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:48:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T20:48:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:53 smithi184 ceph-mon[100622]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:53 smithi184 ceph-mon[102844]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:53.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:53.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:48:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:48:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:48:53.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:48:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:53 smithi196 ceph-mon[76349]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:55 smithi184 ceph-mon[100622]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:55 smithi184 ceph-mon[102844]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:55 smithi196 ceph-mon[76349]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:57 smithi184 ceph-mon[100622]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:57 smithi184 ceph-mon[102844]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:57 smithi196 ceph-mon[76349]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[100622]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[102844]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:59 smithi196 ceph-mon[76349]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:00.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:00.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:00.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:01 smithi184 ceph-mon[100622]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:01 smithi184 ceph-mon[102844]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:01 smithi196 ceph-mon[76349]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:49:03.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:03 smithi184 ceph-mon[100622]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:03.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:03 smithi184 ceph-mon[102844]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:03.818 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:03 smithi196 ceph-mon[76349]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:03.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:03.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:03.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:05 smithi196 ceph-mon[76349]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[100622]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:05 smithi184 ceph-mon[102844]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:07 smithi184 ceph-mon[100622]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:07 smithi184 ceph-mon[102844]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:07 smithi196 ceph-mon[76349]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:09 smithi196 ceph-mon[76349]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:09 smithi184 ceph-mon[100622]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:09 smithi184 ceph-mon[102844]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:10.317Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:10.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:10.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:11.796 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:49:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:11 smithi196 ceph-mon[76349]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:11 smithi184 ceph-mon[102844]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:11 smithi184 ceph-mon[100622]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 65m - - 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 65m - - 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (60m) 8s ago 61m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (61m) 6s ago 61m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (54m) 8s ago 59m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (53m) 6s ago 65m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (54m) 8s ago 66m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (53m) 8s ago 66m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:49:12.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (52m) 6s ago 65m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (52m) 8s ago 65m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (60m) 8s ago 60m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (60m) 6s ago 60m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (51m) 8s ago 64m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (50m) 8s ago 64m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (49m) 8s ago 63m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (49m) 8s ago 63m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (48m) 6s ago 63m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (47m) 6s ago 62m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (46m) 6s ago 62m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (45m) 6s ago 62m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (54m) 6s ago 61m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (45m) 8s ago 59m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:49:12.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (44m) 6s ago 59m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:49:12.324 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (45m) 8s ago 59m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:49:12.324 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (44m) 6s ago 59m 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:49:12.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:49:12.637 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:49:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:12 smithi196 ceph-mon[76349]: from='client.47856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:12.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:12 smithi184 ceph-mon[100622]: from='client.47856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:12.907 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:12 smithi184 ceph-mon[102844]: from='client.47856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:49:12.907 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:49:12.908 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:49:13.219 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:49:13.219 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:13.219 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:13.219 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:13.219 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:13 smithi196 ceph-mon[76349]: from='client.47862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:13 smithi196 ceph-mon[76349]: from='client.47868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:13 smithi196 ceph-mon[76349]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2180207008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:13.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3901972131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[102844]: from='client.47862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[102844]: from='client.47868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[102844]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2180207008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3901972131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[100622]: from='client.47862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[100622]: from='client.47868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[100622]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2180207008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3901972131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:13.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:13.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:13.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:14 smithi196 ceph-mon[76349]: from='client.57845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:14 smithi184 ceph-mon[100622]: from='client.57845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:14 smithi184 ceph-mon[102844]: from='client.57845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:15 smithi196 ceph-mon[76349]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:15 smithi184 ceph-mon[100622]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:15 smithi184 ceph-mon[102844]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:17 smithi196 ceph-mon[76349]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:17.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:17 smithi184 ceph-mon[100622]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:17.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:17 smithi184 ceph-mon[102844]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:19 smithi196 ceph-mon[76349]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:20.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:20.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:19 smithi184 ceph-mon[100622]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:20.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:20.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:19 smithi184 ceph-mon[102844]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:20.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:20.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:20.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:21 smithi196 ceph-mon[76349]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:21 smithi184 ceph-mon[100622]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:21 smithi184 ceph-mon[102844]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:49:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:23 smithi196 ceph-mon[76349]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:23 smithi184 ceph-mon[100622]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:23 smithi184 ceph-mon[102844]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:23.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:23.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:23.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[100622]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:25 smithi184 ceph-mon[102844]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:25.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:25.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:25 smithi196 ceph-mon[76349]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:26.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:26.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:26.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:26.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:26.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:26.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:27.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:27 smithi184 ceph-mon[100622]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:27.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:27 smithi184 ceph-mon[102844]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:27 smithi196 ceph-mon[76349]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[100622]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[102844]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:29 smithi196 ceph-mon[76349]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:30.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:30.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:30.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:31 smithi184 ceph-mon[100622]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:31 smithi184 ceph-mon[102844]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:31 smithi196 ceph-mon[76349]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:32.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:49:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:33 smithi184 ceph-mon[102844]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:33 smithi184 ceph-mon[100622]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:33.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:33.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:33.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:33 smithi196 ceph-mon[76349]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:35 smithi184 ceph-mon[100622]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:35 smithi184 ceph-mon[102844]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:35 smithi196 ceph-mon[76349]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:37.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:37 smithi184 ceph-mon[100622]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:37.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:37 smithi184 ceph-mon[102844]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:37 smithi196 ceph-mon[76349]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:39 smithi184 ceph-mon[100622]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:39 smithi184 ceph-mon[102844]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:39 smithi196 ceph-mon[76349]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:40.319Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:40.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:40.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:41 smithi184 ceph-mon[100622]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:41 smithi184 ceph-mon[102844]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:41 smithi196 ceph-mon[76349]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:49:43.510 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:49:44.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:43 smithi196 ceph-mon[76349]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:44.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 66m - - 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 66m - - 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (60m) 19s ago 61m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (61m) 18s ago 61m 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (54m) 19s ago 59m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (54m) 18s ago 65m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (55m) 19s ago 67m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (53m) 19s ago 67m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (53m) 18s ago 66m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:49:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (53m) 19s ago 65m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (61m) 19s ago 61m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (61m) 18s ago 61m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (52m) 19s ago 65m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (51m) 19s ago 64m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (50m) 19s ago 64m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (49m) 19s ago 64m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (48m) 18s ago 63m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (47m) 18s ago 63m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (46m) 18s ago 62m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (46m) 18s ago 62m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:49:44.044 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (54m) 18s ago 62m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:49:44.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (45m) 19s ago 60m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:49:44.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (45m) 18s ago 60m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:49:44.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (45m) 19s ago 59m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:49:44.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (45m) 18s ago 59m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:49:44.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-mon[100622]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:44.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-mon[102844]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:44.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:43.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:44.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:43.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:44.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:43.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:44.369 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:49:44.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:49:44.369 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:49:44.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:49:44.370 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:49:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:49:44.642 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:49:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:49:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:44.955 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:44.956 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='client.47892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='client.57863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3267871289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='client.47892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='client.57863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:44.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3267871289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:45.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='client.47892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:45.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='client.57863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:45.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:45.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:45.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:45.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3267871289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:46.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:45 smithi196 ceph-mon[76349]: from='client.47904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:45 smithi196 ceph-mon[76349]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:46.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:45 smithi196 ceph-mon[76349]: from='client.57881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3523606890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[102844]: from='client.47904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[102844]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[102844]: from='client.57881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3523606890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[100622]: from='client.47904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[100622]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[100622]: from='client.57881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3523606890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:46 smithi196 ceph-mon[76349]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[102844]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:49:47.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:46 smithi184 ceph-mon[100622]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:49.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:49 smithi184 ceph-mon[100622]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:49.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:49 smithi184 ceph-mon[102844]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:49 smithi196 ceph-mon[76349]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:50.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:50.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:50.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:51 smithi184 ceph-mon[100622]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:51 smithi184 ceph-mon[102844]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:51 smithi196 ceph-mon[76349]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:49:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T20:49:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:53 smithi196 ceph-mon[76349]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:53 smithi184 ceph-mon[100622]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:53 smithi184 ceph-mon[102844]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:53.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:53.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:49:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:49:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:49:53.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:49:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:55 smithi196 ceph-mon[76349]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:55 smithi184 ceph-mon[100622]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:55 smithi184 ceph-mon[102844]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:57 smithi196 ceph-mon[76349]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:57.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:57 smithi184 ceph-mon[100622]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:57.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:57 smithi184 ceph-mon[102844]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:59 smithi196 ceph-mon[76349]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[100622]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:00.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[102844]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:50:00.000+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:00.573 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:00.573 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:00.321Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:00.573 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:00.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:00.573 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:00.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:01 smithi196 ceph-mon[76349]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:01 smithi184 ceph-mon[100622]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:01 smithi184 ceph-mon[102844]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:50:03.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:03 smithi184 ceph-mon[100622]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:03.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:03 smithi184 ceph-mon[102844]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:03 smithi196 ceph-mon[76349]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:03.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:03.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:03.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:05.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.693 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:05 smithi196 ceph-mon[76349]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[100622]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:05.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:05.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:05 smithi184 ceph-mon[102844]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:06 smithi196 ceph-mon[76349]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[100622]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:06 smithi184 ceph-mon[102844]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:09.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:09 smithi184 ceph-mon[100622]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:09.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:09 smithi184 ceph-mon[102844]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:09 smithi196 ceph-mon[76349]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:10.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:10.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:10.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:10.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:10.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:10.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:11 smithi196 ceph-mon[76349]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:11 smithi184 ceph-mon[100622]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:11 smithi184 ceph-mon[102844]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:50:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:13 smithi196 ceph-mon[76349]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:13 smithi184 ceph-mon[100622]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:13 smithi184 ceph-mon[102844]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:13.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:13.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:13.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:15.257 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:50:15.788 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:50:15.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 66m - - 2024-09-20T20:50:15.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 66m - - 2024-09-20T20:50:15.788 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (61m) 11s ago 62m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:50:15.788 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (62m) 9s ago 62m 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (55m) 11s ago 60m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (55m) 9s ago 66m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (55m) 11s ago 67m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (54m) 11s ago 67m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (53m) 9s ago 66m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (53m) 11s ago 66m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (61m) 11s ago 61m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (61m) 9s ago 61m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (52m) 11s ago 65m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (51m) 11s ago 65m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (50m) 11s ago 65m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (50m) 11s ago 64m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (49m) 9s ago 64m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (48m) 9s ago 63m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:50:15.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (47m) 9s ago 63m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (46m) 9s ago 63m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (55m) 9s ago 62m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (46m) 11s ago 60m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (45m) 9s ago 60m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (46m) 11s ago 60m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:50:15.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (45m) 9s ago 60m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:50:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:15 smithi196 ceph-mon[76349]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:15 smithi184 ceph-mon[102844]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:15 smithi184 ceph-mon[100622]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:50:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:50:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:50:16.099 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:50:16.368 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:50:16.368 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:50:16.368 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:50:16.368 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:50:16.369 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:50:16.681 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:16.681 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:16.681 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:16.681 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:16.682 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[100622]: from='client.47928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[100622]: from='client.47934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[100622]: from='client.47940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/148663619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[102844]: from='client.47928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[102844]: from='client.47934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[102844]: from='client.47940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/148663619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:16 smithi196 ceph-mon[76349]: from='client.47928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:16 smithi196 ceph-mon[76349]: from='client.47934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:16 smithi196 ceph-mon[76349]: from='client.47940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/148663619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:17 smithi196 ceph-mon[76349]: from='client.57917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:17 smithi196 ceph-mon[76349]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1511384384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[100622]: from='client.57917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[100622]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1511384384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[102844]: from='client.57917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[102844]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1511384384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:19 smithi196 ceph-mon[76349]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:19 smithi184 ceph-mon[100622]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:19 smithi184 ceph-mon[102844]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:20.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:20.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:20.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:21 smithi196 ceph-mon[76349]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:21 smithi184 ceph-mon[100622]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:21 smithi184 ceph-mon[102844]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:23 smithi196 ceph-mon[76349]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:23 smithi184 ceph-mon[100622]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:23 smithi184 ceph-mon[102844]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:23.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:23.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:23.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:25 smithi196 ceph-mon[76349]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[100622]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[102844]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:27 smithi196 ceph-mon[76349]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[100622]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:27 smithi184 ceph-mon[102844]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[102844]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[100622]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:29 smithi196 ceph-mon[76349]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:30.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:30.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:30.411 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:30.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:30.411 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:30.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:31 smithi196 ceph-mon[76349]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:31.918 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:31 smithi184 ceph-mon[100622]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:31.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:31 smithi184 ceph-mon[102844]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:50:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:33 smithi196 ceph-mon[76349]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:33 smithi184 ceph-mon[100622]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:33 smithi184 ceph-mon[102844]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:33.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:33.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:33.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:35 smithi196 ceph-mon[76349]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:35 smithi184 ceph-mon[100622]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:35 smithi184 ceph-mon[102844]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:37 smithi196 ceph-mon[76349]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:38.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:37 smithi184 ceph-mon[100622]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:38.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:37 smithi184 ceph-mon[102844]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:39 smithi196 ceph-mon[76349]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:39 smithi184 ceph-mon[102844]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:39 smithi184 ceph-mon[100622]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:40.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:40.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:40.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:41 smithi196 ceph-mon[76349]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:41 smithi184 ceph-mon[100622]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:41 smithi184 ceph-mon[102844]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:50:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:43 smithi196 ceph-mon[76349]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:43 smithi184 ceph-mon[100622]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:43 smithi184 ceph-mon[102844]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:43.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:43.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:43.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:45 smithi196 ceph-mon[76349]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[102844]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[100622]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:46.977 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[100622]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:47 smithi184 ceph-mon[102844]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 67m - - 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 67m - - 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (61m) 2s ago 62m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (62m) 1s ago 62m 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (55m) 2s ago 60m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (55m) 1s ago 66m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (56m) 2s ago 68m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (54m) 2s ago 68m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:50:47.509 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (54m) 1s ago 67m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (54m) 2s ago 66m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (62m) 2s ago 62m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (62m) 1s ago 62m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (53m) 2s ago 66m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (52m) 2s ago 65m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (51m) 2s ago 65m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (51m) 2s ago 65m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (49m) 1s ago 64m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (48m) 1s ago 64m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (47m) 1s ago 64m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (47m) 1s ago 63m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (55m) 1s ago 63m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (46m) 2s ago 61m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:50:47.510 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (46m) 1s ago 61m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:50:47.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (46m) 2s ago 60m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:50:47.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (46m) 1s ago 60m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:50:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:50:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:47 smithi196 ceph-mon[76349]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:47.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:50:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:50:47.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:50:47.820 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:50:48.088 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:50:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:50:48.089 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:50:48.398 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:48.398 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:48.398 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:48.398 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:48.398 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[100622]: from='client.47964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[100622]: from='client.47970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[100622]: from='client.57941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3003989812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[102844]: from='client.47964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[102844]: from='client.47970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[102844]: from='client.57941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3003989812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:48 smithi196 ceph-mon[76349]: from='client.47964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:48 smithi196 ceph-mon[76349]: from='client.47970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:48 smithi196 ceph-mon[76349]: from='client.57941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3003989812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[102844]: from='client.57953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3676241077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[102844]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[100622]: from='client.57953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3676241077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:49 smithi184 ceph-mon[100622]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:49 smithi196 ceph-mon[76349]: from='client.57953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3676241077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:49 smithi196 ceph-mon[76349]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:50.686 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:50.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:50.686 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:50.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:50.686 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:50.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:51 smithi184 ceph-mon[102844]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:51 smithi184 ceph-mon[100622]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:51 smithi196 ceph-mon[76349]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:50:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:50:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:53 smithi196 ceph-mon[76349]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:53 smithi184 ceph-mon[100622]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:53 smithi184 ceph-mon[102844]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:53.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:53.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:50:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:50:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:50:53.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:50:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:55 smithi196 ceph-mon[76349]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:55 smithi184 ceph-mon[100622]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:55 smithi184 ceph-mon[102844]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:57 smithi196 ceph-mon[76349]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:58.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:57 smithi184 ceph-mon[100622]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:58.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:57 smithi184 ceph-mon[102844]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:59 smithi196 ceph-mon[76349]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[102844]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[100622]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:00.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:00.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:00.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:01 smithi196 ceph-mon[76349]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:01 smithi184 ceph-mon[100622]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:01 smithi184 ceph-mon[102844]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:51:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:03 smithi196 ceph-mon[76349]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:03 smithi184 ceph-mon[100622]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:03 smithi184 ceph-mon[102844]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:03.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:03.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:03.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:05 smithi196 ceph-mon[76349]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[102844]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[100622]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:07 smithi196 ceph-mon[76349]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[100622]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:07 smithi184 ceph-mon[102844]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:09.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:09 smithi196 ceph-mon[76349]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:09.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:09.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:09 smithi184 ceph-mon[102844]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:09.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:09.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:09 smithi184 ceph-mon[100622]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:10.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:10.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:10.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:11 smithi184 ceph-mon[100622]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:11 smithi184 ceph-mon[102844]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:11 smithi196 ceph-mon[76349]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:12.163 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:51:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:13 smithi196 ceph-mon[76349]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:13 smithi184 ceph-mon[100622]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:14.072 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:13 smithi184 ceph-mon[102844]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:14.072 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:13.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:14.072 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:13.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:14.072 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:13.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:15 smithi196 ceph-mon[76349]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:15 smithi184 ceph-mon[100622]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:15 smithi184 ceph-mon[102844]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:17 smithi196 ceph-mon[76349]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:17 smithi184 ceph-mon[100622]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:17 smithi184 ceph-mon[102844]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:18.695 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:51:19.223 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:51:19.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 68m - - 2024-09-20T20:51:19.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 67m - - 2024-09-20T20:51:19.245 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (62m) 14s ago 63m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:51:19.245 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (63m) 12s ago 63m 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (56m) 14s ago 61m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (56m) 12s ago 67m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (56m) 14s ago 68m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (55m) 14s ago 69m 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (54m) 12s ago 67m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:51:19.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (55m) 14s ago 67m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (62m) 14s ago 62m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (62m) 12s ago 62m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (53m) 14s ago 66m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (52m) 14s ago 66m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (52m) 14s ago 66m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (51m) 14s ago 65m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:51:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (50m) 12s ago 65m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (49m) 12s ago 64m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (48m) 12s ago 64m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (47m) 12s ago 64m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (56m) 12s ago 63m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (47m) 14s ago 61m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:51:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (47m) 12s ago 61m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:51:19.249 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (47m) 14s ago 61m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:51:19.249 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (46m) 12s ago 61m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:51:19.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:51:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:51:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:51:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:51:19.537 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:51:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[102844]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[102844]: from='client.48000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2437522670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[100622]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[100622]: from='client.48000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2437522670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:19.810 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:51:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:51:19.811 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:51:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:19 smithi196 ceph-mon[76349]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:19 smithi196 ceph-mon[76349]: from='client.48000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2437522670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:20.120 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:51:20.120 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:20.120 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:20.120 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:20.120 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:20.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:20.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:20.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:20.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:20 smithi196 ceph-mon[76349]: from='client.48006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:20 smithi196 ceph-mon[76349]: from='client.57977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:20 smithi196 ceph-mon[76349]: from='client.48024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/687919230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[100622]: from='client.48006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[100622]: from='client.57977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[100622]: from='client.48024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/687919230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[102844]: from='client.48006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[102844]: from='client.57977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[102844]: from='client.48024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/687919230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:21 smithi196 ceph-mon[76349]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:21 smithi184 ceph-mon[100622]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:21 smithi184 ceph-mon[102844]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:51:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:23 smithi196 ceph-mon[76349]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:23 smithi184 ceph-mon[100622]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:23 smithi184 ceph-mon[102844]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:23.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:23.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:23.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:25 smithi196 ceph-mon[76349]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[100622]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[102844]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[100622]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:27 smithi184 ceph-mon[102844]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:27.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:27 smithi196 ceph-mon[76349]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:29 smithi196 ceph-mon[76349]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[100622]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[102844]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:30.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:30.325Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:30.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:30.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:30.707 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:30.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:31 smithi184 ceph-mon[100622]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:31 smithi184 ceph-mon[102844]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:31 smithi196 ceph-mon[76349]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:32.098 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:51:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:33 smithi196 ceph-mon[76349]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:33 smithi184 ceph-mon[100622]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:33 smithi184 ceph-mon[102844]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:33.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:33.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:33.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:35 smithi196 ceph-mon[76349]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:35 smithi184 ceph-mon[100622]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:35 smithi184 ceph-mon[102844]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:37 smithi196 ceph-mon[76349]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:37 smithi184 ceph-mon[100622]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:37 smithi184 ceph-mon[102844]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:39 smithi196 ceph-mon[76349]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:39 smithi184 ceph-mon[102844]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:39 smithi184 ceph-mon[100622]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:40.326Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:40.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:40.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:41 smithi196 ceph-mon[76349]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:41 smithi184 ceph-mon[100622]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:41 smithi184 ceph-mon[102844]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:51:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:43 smithi196 ceph-mon[76349]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:43 smithi184 ceph-mon[100622]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:43 smithi184 ceph-mon[102844]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:43.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:43.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:43.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:45 smithi196 ceph-mon[76349]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[102844]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[100622]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:47 smithi196 ceph-mon[76349]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[100622]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[102844]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:48.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:51:49.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:49.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:49 smithi196 ceph-mon[76349]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:49 smithi184 ceph-mon[102844]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:49 smithi184 ceph-mon[100622]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:50.416 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:51:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:50.327Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:50.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:50.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:50.942 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:51:50.942 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 68m - - 2024-09-20T20:51:50.942 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 68m - - 2024-09-20T20:51:50.942 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (62m) 5s ago 63m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (63m) 4s ago 63m 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (56m) 5s ago 61m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (56m) 4s ago 67m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (57m) 5s ago 69m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (55m) 5s ago 69m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (55m) 4s ago 68m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (55m) 5s ago 68m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (63m) 5s ago 63m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (63m) 4s ago 63m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (54m) 5s ago 67m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (53m) 5s ago 67m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (52m) 5s ago 66m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (52m) 5s ago 66m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:51:50.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (50m) 4s ago 65m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (49m) 4s ago 65m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (48m) 4s ago 65m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (48m) 4s ago 64m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (56m) 4s ago 64m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (47m) 5s ago 62m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (47m) 4s ago 62m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (47m) 5s ago 61m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:51:50.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (47m) 4s ago 61m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:51:51.267 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:51:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:51:51.269 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:51:51.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:51:51.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:51:51.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:51:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:51:51.546 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:51:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[102844]: from='client.58001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[102844]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[102844]: from='client.48042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1688920819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[100622]: from='client.58001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[100622]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[100622]: from='client.48042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1688920819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:51.864 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:51:51.864 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:51.864 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:51.864 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:51.864 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:51 smithi196 ceph-mon[76349]: from='client.58001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:51 smithi196 ceph-mon[76349]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:51 smithi196 ceph-mon[76349]: from='client.48042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1688920819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:52.149 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:51:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[100622]: from='client.58013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[100622]: from='client.48060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1547051426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[102844]: from='client.58013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[102844]: from='client.48060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1547051426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:52 smithi196 ceph-mon[76349]: from='client.58013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:52 smithi196 ceph-mon[76349]: from='client.48060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1547051426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:53 smithi184 ceph-mon[100622]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:53 smithi184 ceph-mon[102844]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:53.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:53.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:53.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:51:53.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:51:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:51:53.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:51:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:53 smithi196 ceph-mon[76349]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:55 smithi196 ceph-mon[76349]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:55 smithi184 ceph-mon[100622]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:55 smithi184 ceph-mon[102844]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:57 smithi196 ceph-mon[76349]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:57 smithi184 ceph-mon[100622]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:57 smithi184 ceph-mon[102844]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:59 smithi196 ceph-mon[76349]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[100622]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[102844]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:00.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:00.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:00.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:01 smithi196 ceph-mon[76349]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:01 smithi184 ceph-mon[100622]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:01 smithi184 ceph-mon[102844]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:03 smithi196 ceph-mon[76349]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:03 smithi184 ceph-mon[100622]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:03 smithi184 ceph-mon[102844]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:03.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:03.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:03.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:05 smithi196 ceph-mon[76349]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:05.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[100622]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[102844]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:07 smithi196 ceph-mon[76349]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:08.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[100622]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[102844]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:09 smithi184 ceph-mon[102844]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:09 smithi184 ceph-mon[100622]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:09 smithi196 ceph-mon[76349]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:10.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:10.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:10.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:11.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:11 smithi184 ceph-mon[100622]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:11.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:11 smithi184 ceph-mon[102844]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:11 smithi196 ceph-mon[76349]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:13.919 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-mon[100622]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:13.919 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:13.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-mon[102844]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:13.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:13.920 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:13.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:13.920 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:13.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:13.920 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:13.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:13 smithi196 ceph-mon[76349]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:15 smithi184 ceph-mon[100622]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:15 smithi184 ceph-mon[102844]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:15 smithi196 ceph-mon[76349]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:17 smithi184 ceph-mon[100622]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:17 smithi184 ceph-mon[102844]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:17 smithi196 ceph-mon[76349]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:19 smithi184 ceph-mon[102844]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:20.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:19 smithi184 ceph-mon[100622]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:19 smithi196 ceph-mon[76349]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:20.329Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:20.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:20.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:22.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:21 smithi184 ceph-mon[100622]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:21 smithi184 ceph-mon[102844]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:22.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:21 smithi196 ceph-mon[76349]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:22.161 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:52:22.689 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:52:22.689 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 69m - - 2024-09-20T20:52:22.689 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 68m - - 2024-09-20T20:52:22.689 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (63m) 17s ago 64m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (64m) 15s ago 64m 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (57m) 17s ago 62m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (57m) 15s ago 68m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (58m) 17s ago 69m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (56m) 17s ago 70m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (55m) 15s ago 68m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (56m) 17s ago 68m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (63m) 17s ago 63m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (63m) 15s ago 63m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (54m) 17s ago 67m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (53m) 17s ago 67m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (53m) 17s ago 67m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (52m) 17s ago 66m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:52:22.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (51m) 15s ago 66m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (50m) 15s ago 65m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (49m) 15s ago 65m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (48m) 15s ago 65m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (57m) 15s ago 64m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (48m) 17s ago 62m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (48m) 15s ago 62m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (48m) 17s ago 62m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:52:22.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (48m) 15s ago 62m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:52:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:52:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:52:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:52:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:52:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:52:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:52:23.282 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:52:23.592 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:52:23.593 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:23.593 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:23.593 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: from='client.48072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: from='client.48078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: from='client.48084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2441536438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3856921514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:23.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:23.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:23.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: from='client.48072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: from='client.48078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: from='client.48084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2441536438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:24.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3856921514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: from='client.48072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: from='client.48078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: from='client.48084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2441536438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:24.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3856921514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:24 smithi196 ceph-mon[76349]: from='client.58061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:24 smithi184 ceph-mon[100622]: from='client.58061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:24 smithi184 ceph-mon[102844]: from='client.58061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[100622]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[102844]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:25 smithi196 ceph-mon[76349]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[100622]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[102844]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:27 smithi196 ceph-mon[76349]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:28.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:29.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:29 smithi196 ceph-mon[76349]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:29 smithi184 ceph-mon[102844]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:29 smithi184 ceph-mon[100622]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:30.329Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:30.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:30.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:31 smithi196 ceph-mon[76349]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:31 smithi184 ceph-mon[100622]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:31 smithi184 ceph-mon[102844]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:33.846 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:33 smithi184 ceph-mon[100622]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:33.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:33 smithi184 ceph-mon[102844]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:33.847 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:33.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:33.847 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:33.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:33.847 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:33.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:33 smithi196 ceph-mon[76349]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:35 smithi184 ceph-mon[100622]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:35 smithi184 ceph-mon[102844]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:35 smithi196 ceph-mon[76349]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:37 smithi184 ceph-mon[100622]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:37 smithi184 ceph-mon[102844]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:37 smithi196 ceph-mon[76349]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:39 smithi184 ceph-mon[100622]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:39 smithi184 ceph-mon[102844]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:39 smithi196 ceph-mon[76349]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:40.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:40.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:40.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:41 smithi196 ceph-mon[76349]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:41 smithi184 ceph-mon[100622]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:41 smithi184 ceph-mon[102844]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-mon[100622]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-mon[102844]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:43.765Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:43.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:43.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:43 smithi196 ceph-mon[76349]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:45 smithi196 ceph-mon[76349]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[100622]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[102844]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:47 smithi196 ceph-mon[76349]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[100622]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:48.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[102844]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:48.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:52:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:49 smithi196 ceph-mon[76349]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:49 smithi184 ceph-mon[100622]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:49 smithi184 ceph-mon[102844]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:50.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:50.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:50.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:51 smithi196 ceph-mon[76349]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:51 smithi184 ceph-mon[100622]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:51 smithi184 ceph-mon[102844]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:52:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:52:53.900 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:52:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:53 smithi184 ceph-mon[100622]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:54.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:53 smithi184 ceph-mon[102844]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:53.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:53.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:52:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:52:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:52:53.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:52:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:53 smithi196 ceph-mon[76349]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:54.453 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 69m - - 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 69m - - 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (63m) 8s ago 64m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (64m) 7s ago 64m 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (57m) 8s ago 62m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (57m) 7s ago 68m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (58m) 8s ago 70m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (56m) 8s ago 70m 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (56m) 7s ago 69m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (56m) 8s ago 69m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:52:54.454 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (64m) 8s ago 64m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (64m) 7s ago 64m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (55m) 8s ago 68m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (54m) 8s ago 68m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (53m) 8s ago 67m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (53m) 8s ago 67m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (51m) 7s ago 66m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (50m) 7s ago 66m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (49m) 7s ago 66m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (49m) 7s ago 65m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (58m) 7s ago 65m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (48m) 8s ago 63m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (48m) 7s ago 63m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:52:54.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (48m) 8s ago 63m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:52:54.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (48m) 7s ago 62m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:54.794 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:52:54.795 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:52:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1096832719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1096832719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:55.074 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:52:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:52:55.074 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:52:55.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:52:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1096832719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:55.391 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:52:55.391 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:55.391 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:55.392 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:55.392 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:55 smithi196 ceph-mon[76349]: from='client.58073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:55 smithi196 ceph-mon[76349]: from='client.48114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:55 smithi196 ceph-mon[76349]: from='client.48120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:55 smithi196 ceph-mon[76349]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/171895174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[100622]: from='client.58073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[100622]: from='client.48114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[100622]: from='client.48120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[100622]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/171895174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[102844]: from='client.58073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[102844]: from='client.48114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[102844]: from='client.48120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[102844]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/171895174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:56 smithi196 ceph-mon[76349]: from='client.48132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:56 smithi184 ceph-mon[100622]: from='client.48132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:56 smithi184 ceph-mon[102844]: from='client.48132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:57 smithi196 ceph-mon[76349]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:58.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:57 smithi184 ceph-mon[100622]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:58.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:57 smithi184 ceph-mon[102844]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:52:59 smithi196 ceph-mon[76349]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:52:59 smithi184 ceph-mon[100622]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:52:59 smithi184 ceph-mon[102844]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:00.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:00.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:00.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:01 smithi196 ceph-mon[76349]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:01 smithi184 ceph-mon[100622]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:01 smithi184 ceph-mon[102844]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T20:53:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:03 smithi184 ceph-mon[100622]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:03 smithi184 ceph-mon[102844]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:03.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:03.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:03.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:03 smithi196 ceph-mon[76349]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:04 smithi196 ceph-mon[76349]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:04 smithi184 ceph-mon[100622]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:04 smithi184 ceph-mon[102844]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:07 smithi196 ceph-mon[76349]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[100622]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[102844]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:09 smithi196 ceph-mon[76349]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:09 smithi184 ceph-mon[100622]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:09 smithi184 ceph-mon[102844]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:10.332Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:10.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:10.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:11.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:11 smithi196 ceph-mon[76349]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:11 smithi184 ceph-mon[102844]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:11 smithi184 ceph-mon[100622]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T20:53:13.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:13 smithi184 ceph-mon[100622]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:13.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:13 smithi184 ceph-mon[102844]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:13 smithi196 ceph-mon[76349]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:13.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:13.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:13.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:14.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:14.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:15.867 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:15 smithi184 ceph-mon[100622]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:15.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:15 smithi184 ceph-mon[102844]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:15 smithi196 ceph-mon[76349]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:17 smithi184 ceph-mon[100622]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:17 smithi184 ceph-mon[102844]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:17 smithi196 ceph-mon[76349]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:19 smithi184 ceph-mon[100622]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:19 smithi184 ceph-mon[102844]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:19 smithi196 ceph-mon[76349]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:20.332Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:20.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:20.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:21 smithi184 ceph-mon[100622]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:21 smithi184 ceph-mon[102844]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:21 smithi196 ceph-mon[76349]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:53:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:23 smithi184 ceph-mon[100622]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:23 smithi184 ceph-mon[102844]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:23.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:23.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:23.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:23 smithi196 ceph-mon[76349]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:53:25.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:25 smithi196 ceph-mon[76349]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:25 smithi184 ceph-mon[100622]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:25 smithi184 ceph-mon[102844]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:26.221 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:53:26.221 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 70m - - 2024-09-20T20:53:26.221 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 69m - - 2024-09-20T20:53:26.221 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (64m) 0s ago 65m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:53:26.221 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (65m) 19s ago 65m 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (58m) 0s ago 63m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (58m) 19s ago 69m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (59m) 0s ago 71m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (57m) 0s ago 71m 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (56m) 19s ago 69m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (57m) 0s ago 69m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (65m) 0s ago 64m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (64m) 19s ago 64m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (55m) 0s ago 68m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (54m) 0s ago 68m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (54m) 0s ago 68m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (53m) 0s ago 67m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (52m) 19s ago 67m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:53:26.222 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (51m) 19s ago 67m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (50m) 19s ago 66m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (49m) 19s ago 66m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (58m) 19s ago 65m 65.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (49m) 0s ago 63m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (49m) 19s ago 64m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (49m) 0s ago 63m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:53:26.223 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (49m) 19s ago 63m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:53:26.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:53:26.537 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:53:26.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:53:26.539 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:53:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[100622]: from='client.48144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2917324915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[102844]: from='client.48144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2917324915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:26.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:53:26.812 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:53:26.812 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:53:26.812 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:53:26.812 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:53:26.813 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:53:27.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:26 smithi196 ceph-mon[76349]: from='client.48144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:27.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:27.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:27.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:27.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2917324915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:27.126 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:53:27.126 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:27.126 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='client.48150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='client.48156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='client.48168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1852645303' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='client.48150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='client.48156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='client.48168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1852645303' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='client.48150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='client.48156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='client.48168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1852645303' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:28.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:29 smithi184 ceph-mon[102844]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:29 smithi184 ceph-mon[100622]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:29 smithi196 ceph-mon[76349]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:30.333Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:30.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:30.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:31.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:31 smithi196 ceph-mon[76349]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:31 smithi184 ceph-mon[102844]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:31 smithi184 ceph-mon[100622]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T20:53:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:33 smithi184 ceph-mon[100622]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:33 smithi184 ceph-mon[102844]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:33.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:33.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:33.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:33 smithi196 ceph-mon[76349]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:35 smithi184 ceph-mon[102844]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:35 smithi184 ceph-mon[100622]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:35 smithi196 ceph-mon[76349]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:37 smithi184 ceph-mon[102844]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:37 smithi184 ceph-mon[100622]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:37 smithi196 ceph-mon[76349]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:39 smithi184 ceph-mon[100622]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:39 smithi184 ceph-mon[102844]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:39 smithi196 ceph-mon[76349]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:40.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:40.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:40.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:41 smithi184 ceph-mon[100622]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:41 smithi184 ceph-mon[102844]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T20:53:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:41 smithi196 ceph-mon[76349]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-mon[100622]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-mon[102844]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:43.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:43.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:43.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:43 smithi196 ceph-mon[76349]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:45 smithi184 ceph-mon[100622]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:45 smithi184 ceph-mon[102844]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:45 smithi196 ceph-mon[76349]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:46 smithi196 ceph-mon[76349]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[100622]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:46 smithi184 ceph-mon[102844]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:48.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:49 smithi196 ceph-mon[76349]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:49 smithi184 ceph-mon[100622]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:49 smithi184 ceph-mon[102844]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:50.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:50.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:50.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:51 smithi196 ceph-mon[76349]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:51 smithi184 ceph-mon[100622]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:52.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:51 smithi184 ceph-mon[102844]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:53:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:53:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:53 smithi196 ceph-mon[76349]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:53 smithi184 ceph-mon[100622]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:53 smithi184 ceph-mon[102844]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:53.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:53.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:53:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:53:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:53:53.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:53:55.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:55 smithi184 ceph-mon[100622]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:55.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:55 smithi184 ceph-mon[102844]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:55 smithi196 ceph-mon[76349]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:57.413 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:53:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:57 smithi196 ceph-mon[76349]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:57.939 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:53:57.939 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 70m - - 2024-09-20T20:53:57.939 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 70m - - 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (64m) 12s ago 65m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (65m) 10s ago 65m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (58m) 12s ago 63m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (58m) 10s ago 69m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (59m) 12s ago 71m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (57m) 12s ago 71m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (57m) 10s ago 70m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (57m) 12s ago 70m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (65m) 12s ago 65m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (65m) 10s ago 65m 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (56m) 12s ago 69m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (55m) 12s ago 69m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:53:57.940 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (54m) 12s ago 68m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (54m) 12s ago 68m 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (52m) 10s ago 67m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (52m) 10s ago 67m 62.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (50m) 10s ago 67m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (50m) 10s ago 66m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (59m) 10s ago 66m 66.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (49m) 12s ago 64m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (49m) 10s ago 64m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (49m) 12s ago 64m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:53:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (49m) 10s ago 64m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:53:57.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:57 smithi184 ceph-mon[100622]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:57.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:57 smithi184 ceph-mon[102844]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:58.256 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:53:58.257 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:53:58.526 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:53:58.526 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:53:58.527 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:53:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[100622]: from='client.58145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[100622]: from='client.48186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2076201649' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[102844]: from='client.58145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[102844]: from='client.48186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2076201649' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:58.844 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:53:58.844 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:58.844 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:58.844 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:58.844 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:58 smithi196 ceph-mon[76349]: from='client.58145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:58 smithi196 ceph-mon[76349]: from='client.48186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2076201649' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: from='client.48192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: from='client.48204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2526772745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: from='client.48192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: from='client.48204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:00.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:00.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:53:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2526772745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: from='client.48192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: from='client.48204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:53:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2526772745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:00.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:00.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:00.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:01 smithi184 ceph-mon[100622]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:02.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:01 smithi184 ceph-mon[102844]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:01 smithi196 ceph-mon[76349]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:54:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:03 smithi184 ceph-mon[100622]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:03 smithi184 ceph-mon[102844]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:03.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:03.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:03.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:03 smithi196 ceph-mon[76349]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:06.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:05 smithi184 ceph-mon[100622]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:54:06.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:05 smithi184 ceph-mon[102844]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:54:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:05 smithi196 ceph-mon[76349]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:54:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[100622]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:06 smithi184 ceph-mon[102844]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T20:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:06 smithi196 ceph-mon[76349]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:09 smithi184 ceph-mon[100622]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T20:54:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:09 smithi184 ceph-mon[102844]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T20:54:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:09 smithi196 ceph-mon[76349]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T20:54:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:10.336Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:10.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:10.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:11 smithi196 ceph-mon[76349]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:54:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:11 smithi184 ceph-mon[100622]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:54:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:11 smithi184 ceph-mon[102844]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:54:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:54:13.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:13 smithi196 ceph-mon[76349]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T20:54:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:13 smithi184 ceph-mon[102844]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T20:54:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:13 smithi184 ceph-mon[100622]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T20:54:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:13.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:13.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:13.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:16.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:15 smithi184 ceph-mon[100622]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T20:54:16.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:15 smithi184 ceph-mon[102844]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T20:54:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:15 smithi196 ceph-mon[76349]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T20:54:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:17 smithi184 ceph-mon[102844]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:17 smithi184 ceph-mon[100622]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:17 smithi196 ceph-mon[76349]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:19 smithi184 ceph-mon[102844]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:19 smithi184 ceph-mon[100622]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:19 smithi196 ceph-mon[76349]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:54:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:20.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:20.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:20.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:21 smithi184 ceph-mon[100622]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:54:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:21 smithi184 ceph-mon[102844]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:54:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:21 smithi196 ceph-mon[76349]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T20:54:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:54:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:23 smithi184 ceph-mon[100622]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:54:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:23 smithi184 ceph-mon[102844]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:54:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:23.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:23.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:23.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:23 smithi196 ceph-mon[76349]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T20:54:25.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:25 smithi196 ceph-mon[76349]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:54:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:25 smithi184 ceph-mon[100622]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:54:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:25 smithi184 ceph-mon[102844]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:54:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:27 smithi196 ceph-mon[76349]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[100622]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:27 smithi184 ceph-mon[102844]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T20:54:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:29.138 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 71m - - 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 70m - - 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (65m) 3s ago 66m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (66m) 2s ago 66m 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:54:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (59m) 3s ago 64m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (59m) 2s ago 70m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (60m) 3s ago 72m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (58m) 3s ago 72m 160M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (57m) 2s ago 70m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (58m) 3s ago 70m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (66m) 3s ago 66m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:54:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (65m) 2s ago 65m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (56m) 3s ago 70m 67.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (55m) 3s ago 69m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (55m) 3s ago 69m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (54m) 3s ago 68m 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (53m) 2s ago 68m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (52m) 2s ago 68m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (51m) 2s ago 67m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:54:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (50m) 2s ago 67m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:54:29.669 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (59m) 2s ago 66m 67.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:54:29.669 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (50m) 3s ago 64m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:54:29.669 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (50m) 2s ago 65m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:54:29.669 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (50m) 3s ago 64m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:54:29.669 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (50m) 2s ago 64m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:54:29.974 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:54:29.975 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:54:29.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:29.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[100622]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:54:29.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:29.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:29.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[102844]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:54:29.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:29 smithi196 ceph-mon[76349]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T20:54:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:30.243 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:54:30.243 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:54:30.243 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:54:30.244 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:54:30.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:30.338Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:30.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:30.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:30.556 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:54:30.556 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:30.557 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:30.557 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:30.557 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[100622]: from='client.48216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[100622]: from='client.58187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[100622]: from='client.58193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/599810980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2414107508' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[102844]: from='client.48216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[102844]: from='client.58187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[102844]: from='client.58193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/599810980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2414107508' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:30 smithi196 ceph-mon[76349]: from='client.48216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:30 smithi196 ceph-mon[76349]: from='client.58187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:30 smithi196 ceph-mon[76349]: from='client.58193 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/599810980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:31.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2414107508' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:31 smithi184 ceph-mon[100622]: from='client.48240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:31 smithi184 ceph-mon[100622]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:54:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:31 smithi184 ceph-mon[102844]: from='client.48240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:32.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:31 smithi184 ceph-mon[102844]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:54:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:31 smithi196 ceph-mon[76349]: from='client.48240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:31 smithi196 ceph-mon[76349]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T20:54:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:54:33.776 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:33 smithi184 ceph-mon[100622]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T20:54:33.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:33 smithi184 ceph-mon[102844]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T20:54:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:33.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:33.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:33.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:33 smithi196 ceph-mon[76349]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T20:54:35.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:35 smithi184 ceph-mon[100622]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T20:54:35.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:35 smithi184 ceph-mon[102844]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T20:54:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:35 smithi196 ceph-mon[76349]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T20:54:37.822 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:37 smithi184 ceph-mon[100622]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T20:54:37.822 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:37 smithi184 ceph-mon[102844]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T20:54:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:37 smithi196 ceph-mon[76349]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T20:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:39 smithi184 ceph-mon[102844]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:54:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:39 smithi184 ceph-mon[100622]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:39 smithi196 ceph-mon[76349]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T20:54:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:40.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:40.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:40.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:41 smithi184 ceph-mon[100622]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T20:54:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:41 smithi184 ceph-mon[102844]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T20:54:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:41 smithi196 ceph-mon[76349]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T20:54:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:54:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:43 smithi184 ceph-mon[100622]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:43 smithi184 ceph-mon[102844]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:43.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:43.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:43.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:43 smithi196 ceph-mon[76349]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:45 smithi184 ceph-mon[100622]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:45 smithi184 ceph-mon[102844]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:45 smithi196 ceph-mon[76349]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[100622]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:47 smithi184 ceph-mon[102844]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:47 smithi196 ceph-mon[76349]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:54:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:49 smithi184 ceph-mon[100622]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:49 smithi184 ceph-mon[102844]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:49 smithi196 ceph-mon[76349]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:50.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:50.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:50.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:52.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:51 smithi184 ceph-mon[100622]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:52.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:51 smithi184 ceph-mon[102844]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:52.050 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:54:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:54:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:51 smithi196 ceph-mon[76349]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:53 smithi196 ceph-mon[76349]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:53 smithi184 ceph-mon[100622]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:53 smithi184 ceph-mon[102844]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:53.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:53.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:54:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:54:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:54:53.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:54:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:55 smithi184 ceph-mon[100622]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:55 smithi184 ceph-mon[102844]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:56.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:55 smithi196 ceph-mon[76349]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:57.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:57 smithi184 ceph-mon[100622]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:57.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:57 smithi184 ceph-mon[102844]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:57 smithi196 ceph-mon[76349]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:54:59 smithi184 ceph-mon[100622]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:54:59 smithi184 ceph-mon[102844]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:54:59 smithi196 ceph-mon[76349]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:00.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:00.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:00.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:00.854 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:55:01.380 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:55:01.380 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 71m - - 2024-09-20T20:55:01.380 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 71m - - 2024-09-20T20:55:01.380 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (65m) 15s ago 67m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:55:01.380 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (66m) 13s ago 66m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (59m) 15s ago 65m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (59m) 13s ago 70m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (60m) 15s ago 72m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (59m) 15s ago 72m 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (58m) 13s ago 71m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (58m) 15s ago 71m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (66m) 15s ago 66m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (66m) 13s ago 66m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (57m) 15s ago 70m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (56m) 15s ago 70m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (55m) 15s ago 69m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (55m) 15s ago 69m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (53m) 13s ago 69m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:55:01.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (53m) 13s ago 68m 63.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (51m) 13s ago 68m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (51m) 13s ago 67m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (60m) 13s ago 67m 67.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (50m) 15s ago 65m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (50m) 13s ago 65m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (50m) 15s ago 65m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:55:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (50m) 13s ago 65m 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:55:01.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:55:01.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:01.693 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:55:01.693 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:55:01.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:55:01.693 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:55:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:55:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:55:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:55:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:55:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:55:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:55:01.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[100622]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:01.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[100622]: from='client.58217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:01.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1446687765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:01.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[102844]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:01.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[102844]: from='client.58217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:01.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1446687765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:01 smithi196 ceph-mon[76349]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:01 smithi196 ceph-mon[76349]: from='client.58217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1446687765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:02.271 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:55:02.271 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:02.271 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:02.271 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:02.271 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:02.271 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:55:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[102844]: from='client.48258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[102844]: from='client.58229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1800941986' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[100622]: from='client.48258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[100622]: from='client.58229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1800941986' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:02 smithi196 ceph-mon[76349]: from='client.48258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:02 smithi196 ceph-mon[76349]: from='client.58229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1800941986' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-mon[100622]: from='client.58241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-mon[100622]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-mon[102844]: from='client.58241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-mon[102844]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:03.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:03.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:03.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:03 smithi196 ceph-mon[76349]: from='client.58241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:03 smithi196 ceph-mon[76349]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:05 smithi196 ceph-mon[76349]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:05 smithi184 ceph-mon[100622]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:05 smithi184 ceph-mon[102844]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:07 smithi196 ceph-mon[76349]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[102844]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:07 smithi184 ceph-mon[100622]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:08 smithi196 ceph-mon[76349]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[100622]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:08 smithi184 ceph-mon[102844]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:10.341Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:10.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:10.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:11 smithi184 ceph-mon[102844]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:11 smithi184 ceph-mon[100622]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:11 smithi196 ceph-mon[76349]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:55:13.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:13 smithi184 ceph-mon[100622]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:13.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:13 smithi184 ceph-mon[102844]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:13 smithi196 ceph-mon[76349]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:13.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:13.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:13.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:14.886 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:15 smithi184 ceph-mon[100622]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:15 smithi184 ceph-mon[102844]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:15 smithi196 ceph-mon[76349]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:17 smithi184 ceph-mon[100622]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:17 smithi184 ceph-mon[102844]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:17 smithi196 ceph-mon[76349]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:19 smithi184 ceph-mon[102844]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:19 smithi184 ceph-mon[100622]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:19 smithi196 ceph-mon[76349]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:20.341Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:20.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:20.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:21 smithi184 ceph-mon[100622]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:21 smithi184 ceph-mon[102844]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:21 smithi196 ceph-mon[76349]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:55:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:23 smithi184 ceph-mon[100622]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:23 smithi184 ceph-mon[102844]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:23.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:23.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:23.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:23 smithi196 ceph-mon[76349]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:25.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:25 smithi196 ceph-mon[76349]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:25 smithi184 ceph-mon[100622]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:25 smithi184 ceph-mon[102844]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[100622]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:27 smithi184 ceph-mon[102844]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:27 smithi196 ceph-mon[76349]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:30.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:30.808 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:30.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:30.809 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:30.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:31 smithi184 ceph-mon[100622]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:32.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:31 smithi184 ceph-mon[102844]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:31 smithi196 ceph-mon[76349]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:55:32.583 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:55:33.131 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:55:33.131 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 72m - - 2024-09-20T20:55:33.131 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 72m - - 2024-09-20T20:55:33.131 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (66m) 6s ago 67m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (67m) 5s ago 67m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (60m) 6s ago 65m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (60m) 5s ago 71m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (61m) 6s ago 73m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (59m) 6s ago 73m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (58m) 5s ago 71m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (59m) 6s ago 71m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (67m) 6s ago 67m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (67m) 5s ago 67m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (57m) 6s ago 71m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (56m) 6s ago 70m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (56m) 6s ago 70m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (55m) 6s ago 69m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (54m) 5s ago 69m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:55:33.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (53m) 5s ago 69m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (52m) 5s ago 68m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (51m) 5s ago 68m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (60m) 5s ago 67m 64.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (51m) 6s ago 66m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (51m) 5s ago 66m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (51m) 6s ago 65m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:55:33.133 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (51m) 5s ago 65m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:55:33.459 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:55:33.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[102844]: from='client.58253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:33.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[102844]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:33.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[102844]: from='client.58259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:33.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1150458762' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:33.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[100622]: from='client.58253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:33.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[100622]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:33.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[100622]: from='client.58259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:33.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1150458762' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:55:33.731 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:55:33.732 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:55:34.040 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:55:34.040 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:34.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:34.040 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:34.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:34.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:33.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:34.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:33.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:34.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:33.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:33 smithi196 ceph-mon[76349]: from='client.58253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:33 smithi196 ceph-mon[76349]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:33 smithi196 ceph-mon[76349]: from='client.58259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1150458762' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:34.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:34 smithi196 ceph-mon[76349]: from='client.48300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:34.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:34 smithi196 ceph-mon[76349]: from='client.48312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:34.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/294005121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[102844]: from='client.48300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[102844]: from='client.48312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/294005121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[100622]: from='client.48300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[100622]: from='client.48312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/294005121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:35.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:35 smithi196 ceph-mon[76349]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:35 smithi184 ceph-mon[100622]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:35 smithi184 ceph-mon[102844]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:37.927 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:37 smithi184 ceph-mon[100622]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:37.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:37 smithi184 ceph-mon[102844]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:37 smithi196 ceph-mon[76349]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:39 smithi184 ceph-mon[100622]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:39 smithi184 ceph-mon[102844]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:39 smithi196 ceph-mon[76349]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:40.407 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:40.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:40.407 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:40.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:40.407 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:40.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:41 smithi184 ceph-mon[100622]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:41 smithi184 ceph-mon[102844]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:41 smithi196 ceph-mon[76349]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:55:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:43 smithi184 ceph-mon[100622]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:43 smithi184 ceph-mon[102844]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:43.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:43.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:43.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:43 smithi196 ceph-mon[76349]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:45 smithi184 ceph-mon[100622]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:45 smithi184 ceph-mon[102844]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:45 smithi196 ceph-mon[76349]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:47 smithi196 ceph-mon[76349]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[100622]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[102844]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[102844]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:49 smithi184 ceph-mon[100622]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:49 smithi196 ceph-mon[76349]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:50.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:50.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:50.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:51 smithi184 ceph-mon[100622]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:52.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:51 smithi184 ceph-mon[102844]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:51 smithi196 ceph-mon[76349]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:55:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:55:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:53 smithi184 ceph-mon[100622]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:53 smithi184 ceph-mon[102844]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:53.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:53.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:55:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:55:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:55:53.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:55:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:53 smithi196 ceph-mon[76349]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:55.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:55 smithi196 ceph-mon[76349]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:55 smithi184 ceph-mon[100622]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:55 smithi184 ceph-mon[102844]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:57 smithi184 ceph-mon[100622]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:57 smithi184 ceph-mon[102844]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:58.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:57 smithi196 ceph-mon[76349]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:59.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:59.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[100622]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:59.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:59.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:59.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[102844]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:59.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:59 smithi196 ceph-mon[76349]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:00.479 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:00.345Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:00.479 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:00.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:00.479 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:00.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:01 smithi184 ceph-mon[100622]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:01 smithi184 ceph-mon[102844]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:01 smithi196 ceph-mon[76349]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:56:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:03 smithi184 ceph-mon[100622]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:03 smithi184 ceph-mon[102844]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:03.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:03.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:03.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:03 smithi196 ceph-mon[76349]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:04.337 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 72m - - 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 72m - - 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (66m) 18s ago 68m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (67m) 16s ago 67m 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (61m) 18s ago 66m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (60m) 16s ago 71m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (61m) 18s ago 73m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (60m) 18s ago 73m 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (59m) 16s ago 72m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (59m) 18s ago 72m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:56:04.863 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (67m) 18s ago 67m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (67m) 16s ago 67m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (58m) 18s ago 71m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (57m) 18s ago 71m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (56m) 18s ago 70m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (56m) 18s ago 70m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (55m) 16s ago 70m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (54m) 16s ago 69m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (53m) 16s ago 69m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (52m) 16s ago 68m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (61m) 16s ago 68m 64.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (52m) 18s ago 66m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (51m) 16s ago 66m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (51m) 18s ago 66m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:56:04.864 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (51m) 16s ago 66m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:56:05.171 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:56:05.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:56:05.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:56:05.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:56:05.172 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:56:05.439 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:56:05.440 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:56:05.440 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:56:05.752 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:56:05.752 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:05.753 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:05.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[100622]: from='client.48324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[100622]: from='client.48330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[100622]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[100622]: from='client.48336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3184458017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[102844]: from='client.48324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[102844]: from='client.48330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[102844]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[102844]: from='client.48336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3184458017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:05 smithi196 ceph-mon[76349]: from='client.48324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:05 smithi196 ceph-mon[76349]: from='client.48330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:05 smithi196 ceph-mon[76349]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:05 smithi196 ceph-mon[76349]: from='client.48336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:06.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3184458017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:06 smithi184 ceph-mon[100622]: from='client.58313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1332333044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:06 smithi184 ceph-mon[102844]: from='client.58313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1332333044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:06 smithi196 ceph-mon[76349]: from='client.58313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1332333044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[100622]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[102844]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:07 smithi196 ceph-mon[76349]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[100622]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:09 smithi184 ceph-mon[102844]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:09.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:09.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:09 smithi196 ceph-mon[76349]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:10.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:10.346Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:10.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:10.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:10.802 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:10.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:11 smithi184 ceph-mon[102844]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:11 smithi184 ceph-mon[100622]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:11 smithi196 ceph-mon[76349]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:56:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:13 smithi184 ceph-mon[102844]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:13 smithi184 ceph-mon[100622]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:13.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:13.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:13.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:13 smithi196 ceph-mon[76349]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:15.766 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:15 smithi196 ceph-mon[76349]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:15 smithi184 ceph-mon[100622]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:15 smithi184 ceph-mon[102844]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:17.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:17 smithi196 ceph-mon[76349]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:17 smithi184 ceph-mon[100622]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:17 smithi184 ceph-mon[102844]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:19 smithi184 ceph-mon[100622]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:19 smithi184 ceph-mon[102844]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:19 smithi196 ceph-mon[76349]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:20.592 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:20.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:20.592 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:20.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:20.592 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:20.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:21 smithi184 ceph-mon[100622]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:21 smithi184 ceph-mon[102844]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:21 smithi196 ceph-mon[76349]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T20:56:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:23 smithi184 ceph-mon[100622]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:23 smithi184 ceph-mon[102844]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:23.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:23.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:23.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:23 smithi196 ceph-mon[76349]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:25.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:25 smithi196 ceph-mon[76349]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:25 smithi184 ceph-mon[100622]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:25 smithi184 ceph-mon[102844]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[100622]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[102844]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:27 smithi196 ceph-mon[76349]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:30.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:30.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:30.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:31 smithi184 ceph-mon[100622]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:31 smithi184 ceph-mon[102844]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:31 smithi196 ceph-mon[76349]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:56:33.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:33 smithi184 ceph-mon[100622]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:33.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:33 smithi184 ceph-mon[102844]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:33.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:33.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:33.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:33 smithi196 ceph-mon[76349]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:36.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:35 smithi196 ceph-mon[76349]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:36.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:35 smithi184 ceph-mon[100622]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:36.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:35 smithi184 ceph-mon[102844]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:36.047 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 73m - - 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 73m - - 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (67m) 9s ago 68m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (68m) 8s ago 68m 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (61m) 9s ago 66m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (61m) 8s ago 72m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:56:36.573 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (62m) 9s ago 74m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (60m) 9s ago 74m 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (60m) 8s ago 72m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (60m) 9s ago 72m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (68m) 9s ago 68m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (68m) 8s ago 68m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (58m) 9s ago 72m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (58m) 9s ago 71m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (57m) 9s ago 71m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:56:36.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (56m) 9s ago 70m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (55m) 8s ago 70m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (54m) 8s ago 70m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (53m) 8s ago 69m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (52m) 8s ago 69m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (61m) 8s ago 68m 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (52m) 9s ago 67m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (52m) 8s ago 67m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (52m) 9s ago 66m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:56:36.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (52m) 8s ago 66m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:56:36.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:56:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:56:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:56:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:56:36.882 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:56:37.151 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:56:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:56:37.177 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:56:37.461 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:56:37.461 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:37.461 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:37.462 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:37.462 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: from='client.48360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: from='client.58331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: from='client.58337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/217097818' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3318510827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: from='client.48360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: from='client.58331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: from='client.58337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/217097818' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3318510827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: from='client.48360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: from='client.58331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: from='client.58337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/217097818' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:38.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3318510827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:38.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:38 smithi196 ceph-mon[76349]: from='client.58349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:38 smithi184 ceph-mon[100622]: from='client.58349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:38 smithi184 ceph-mon[102844]: from='client.58349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:39.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:39.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:39 smithi184 ceph-mon[100622]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:39.972 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:39.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:39 smithi184 ceph-mon[102844]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:40.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:40.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:39 smithi196 ceph-mon[76349]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:40.348Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:40.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:40.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:41.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:41 smithi184 ceph-mon[100622]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:41.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:41 smithi184 ceph-mon[102844]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:41 smithi196 ceph-mon[76349]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:42.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:56:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:43 smithi184 ceph-mon[100622]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:43 smithi184 ceph-mon[102844]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:43.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:43.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:43.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:43 smithi196 ceph-mon[76349]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:45 smithi184 ceph-mon[100622]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:45 smithi184 ceph-mon[102844]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:45 smithi196 ceph-mon[76349]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[102844]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[100622]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:47 smithi196 ceph-mon[76349]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[102844]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:49 smithi184 ceph-mon[100622]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:56:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:49 smithi196 ceph-mon[76349]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:50.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:50.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:50.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:51 smithi184 ceph-mon[100622]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:51 smithi184 ceph-mon[102844]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:51 smithi196 ceph-mon[76349]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:56:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T20:56:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:53 smithi184 ceph-mon[100622]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:53 smithi184 ceph-mon[102844]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:53.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:53.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:56:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:56:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:56:53.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:56:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:53 smithi196 ceph-mon[76349]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:55 smithi184 ceph-mon[100622]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:55 smithi184 ceph-mon[102844]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:55 smithi196 ceph-mon[76349]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:58.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:57 smithi196 ceph-mon[76349]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:57 smithi184 ceph-mon[100622]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:57 smithi184 ceph-mon[102844]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[100622]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[102844]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:59 smithi196 ceph-mon[76349]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:00.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:00.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:00.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:00.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:00.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:00.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:01.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:01 smithi184 ceph-mon[100622]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:01.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:01 smithi184 ceph-mon[102844]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:01 smithi196 ceph-mon[76349]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:57:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:03 smithi184 ceph-mon[100622]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:03 smithi184 ceph-mon[102844]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:03.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:03.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:03.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:03 smithi196 ceph-mon[76349]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:05 smithi184 ceph-mon[100622]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:05 smithi184 ceph-mon[102844]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:05 smithi196 ceph-mon[76349]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:07.752 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:57:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[100622]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[102844]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:07 smithi196 ceph-mon[76349]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:57:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 73m - - 2024-09-20T20:57:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 73m - - 2024-09-20T20:57:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (68m) 1s ago 69m 30.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:57:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (68m) 19s ago 68m 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (62m) 1s ago 67m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (61m) 19s ago 72m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (62m) 1s ago 74m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (61m) 1s ago 74m 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (60m) 19s ago 73m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (60m) 1s ago 73m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (68m) 1s ago 68m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (68m) 19s ago 68m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (59m) 1s ago 72m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (58m) 1s ago 72m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (57m) 1s ago 71m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (57m) 1s ago 71m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (56m) 19s ago 71m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:57:08.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (55m) 19s ago 70m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (54m) 19s ago 70m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (53m) 19s ago 69m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (62m) 19s ago 69m 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (53m) 1s ago 67m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (52m) 19s ago 67m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (52m) 1s ago 67m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:57:08.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (52m) 19s ago 67m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:57:08.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:57:08.592 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:57:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:57:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:57:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:57:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:57:08.865 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:57:08.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[100622]: from='client.58361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:08.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1745967751' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:08.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[102844]: from='client.58361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:08.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1745967751' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:08.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:08.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:08 smithi196 ceph-mon[76349]: from='client.58361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1745967751' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:09.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:09.174 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:57:09.174 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:09.174 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:09.174 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:09.174 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: from='client.58367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: from='client.58373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: from='client.48420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2809903062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: from='client.58367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: from='client.58373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:10.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:10.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: from='client.48420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2809903062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: from='client.58367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: from='client.58373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: from='client.48420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2809903062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:10.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:10.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:10.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:11 smithi184 ceph-mon[102844]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:11 smithi184 ceph-mon[100622]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:11 smithi196 ceph-mon[76349]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:57:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:13 smithi184 ceph-mon[100622]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:13 smithi184 ceph-mon[102844]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:13.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:13.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:13.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:13 smithi196 ceph-mon[76349]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:14.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:14.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:15 smithi184 ceph-mon[100622]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:15 smithi184 ceph-mon[102844]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:15 smithi196 ceph-mon[76349]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:17.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:17 smithi196 ceph-mon[76349]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:17 smithi184 ceph-mon[100622]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:17 smithi184 ceph-mon[102844]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:19 smithi184 ceph-mon[100622]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:19 smithi184 ceph-mon[102844]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:19 smithi196 ceph-mon[76349]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:20.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:20.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:20.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:21 smithi184 ceph-mon[100622]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:21 smithi184 ceph-mon[102844]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:21 smithi196 ceph-mon[76349]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:22.170 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T20:57:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:23 smithi184 ceph-mon[100622]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:23 smithi184 ceph-mon[102844]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:23.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:23.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:23.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:23 smithi196 ceph-mon[76349]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:25 smithi184 ceph-mon[102844]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:25 smithi184 ceph-mon[100622]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:25 smithi196 ceph-mon[76349]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[100622]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[102844]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:27 smithi196 ceph-mon[76349]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[100622]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[102844]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:29 smithi196 ceph-mon[76349]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:30.352Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:30.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:30.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:31 smithi184 ceph-mon[100622]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:31 smithi184 ceph-mon[102844]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:57:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:31 smithi196 ceph-mon[76349]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:33 smithi184 ceph-mon[100622]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:33 smithi184 ceph-mon[102844]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:33.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:33.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:33.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:33 smithi196 ceph-mon[76349]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:35 smithi184 ceph-mon[100622]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:35 smithi184 ceph-mon[102844]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:35 smithi196 ceph-mon[76349]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:38.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:37 smithi196 ceph-mon[76349]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:37 smithi184 ceph-mon[100622]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:37 smithi184 ceph-mon[102844]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:39.471 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:57:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:39 smithi196 ceph-mon[76349]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 74m - - 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 74m - - 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (68m) 12s ago 69m 30.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (69m) 11s ago 69m 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (62m) 12s ago 67m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:57:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (62m) 11s ago 73m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (63m) 12s ago 75m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (61m) 12s ago 75m 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (61m) 11s ago 73m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (61m) 12s ago 73m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (69m) 12s ago 69m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (69m) 11s ago 69m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (59m) 12s ago 73m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (59m) 12s ago 72m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (58m) 12s ago 72m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (57m) 12s ago 72m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (56m) 11s ago 71m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (55m) 11s ago 71m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (54m) 11s ago 70m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:57:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (53m) 11s ago 70m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:57:39.996 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (62m) 11s ago 70m 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:57:39.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (53m) 12s ago 68m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:57:39.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (53m) 11s ago 68m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:57:39.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (53m) 12s ago 67m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:57:39.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (53m) 11s ago 67m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:57:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:39 smithi184 ceph-mon[102844]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:39 smithi184 ceph-mon[100622]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:40.305 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:57:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:57:40.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:57:40.307 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:57:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:57:40.577 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:57:40.577 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:57:40.577 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:57:40.577 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:57:40.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:40.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:40.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:40.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:40.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:40.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:40.885 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:57:40.885 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:40.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:40.885 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:40.886 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[100622]: from='client.58397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[100622]: from='client.58403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2656773135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[102844]: from='client.58397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[102844]: from='client.58403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:40.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2656773135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:40 smithi196 ceph-mon[76349]: from='client.58397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:40 smithi196 ceph-mon[76349]: from='client.58403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2656773135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[100622]: from='client.58409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[100622]: from='client.48456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[100622]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1610590462' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:57:42.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[102844]: from='client.58409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[102844]: from='client.48456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[102844]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:42.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1610590462' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:41 smithi196 ceph-mon[76349]: from='client.58409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:41 smithi196 ceph-mon[76349]: from='client.48456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:41 smithi196 ceph-mon[76349]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1610590462' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:44.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-mon[100622]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:44.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:44.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-mon[102844]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:44.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:43.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:43.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:44.037 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:43.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:43 smithi196 ceph-mon[76349]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:45 smithi196 ceph-mon[76349]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:45 smithi184 ceph-mon[100622]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:45 smithi184 ceph-mon[102844]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:47 smithi196 ceph-mon[76349]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[100622]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[102844]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:49 smithi196 ceph-mon[76349]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[100622]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[102844]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:57:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:50.354Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:50.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:50.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:51 smithi196 ceph-mon[76349]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:51 smithi184 ceph-mon[100622]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:57:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:57:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:51 smithi184 ceph-mon[102844]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:53 smithi184 ceph-mon[100622]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:53 smithi184 ceph-mon[102844]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:53.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:53.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:57:54.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:57:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:57:53.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:57:54.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:53 smithi196 ceph-mon[76349]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:55 smithi196 ceph-mon[76349]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:55 smithi184 ceph-mon[100622]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:55 smithi184 ceph-mon[102844]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:57 smithi196 ceph-mon[76349]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:57 smithi184 ceph-mon[100622]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:57 smithi184 ceph-mon[102844]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:57:59 smithi196 ceph-mon[76349]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:57:59 smithi184 ceph-mon[100622]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:57:59 smithi184 ceph-mon[102844]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:00.354Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:00.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:00.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:01 smithi196 ceph-mon[76349]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:02.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:01 smithi184 ceph-mon[100622]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:02.167 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:01 smithi184 ceph-mon[102844]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:02.167 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:58:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:03 smithi184 ceph-mon[100622]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:03 smithi184 ceph-mon[102844]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:03.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:03.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:03.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:03 smithi196 ceph-mon[76349]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:05 smithi196 ceph-mon[76349]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:05 smithi184 ceph-mon[102844]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:05 smithi184 ceph-mon[100622]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:07 smithi196 ceph-mon[76349]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[100622]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[102844]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:09 smithi196 ceph-mon[76349]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:10.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[102844]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[100622]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:10.355Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:10.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:10.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:11.182 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 74m - - 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 74m - - 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (69m) 4s ago 70m 30.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (70m) 2s ago 69m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (63m) 4s ago 68m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (62m) 2s ago 74m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (63m) 4s ago 75m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (62m) 4s ago 75m 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (61m) 2s ago 74m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (61m) 4s ago 74m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:58:11.722 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (69m) 4s ago 69m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (69m) 2s ago 69m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (60m) 4s ago 73m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (59m) 4s ago 73m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (58m) 4s ago 72m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (58m) 4s ago 72m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (57m) 2s ago 72m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (56m) 2s ago 71m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (55m) 2s ago 71m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (54m) 2s ago 71m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (63m) 2s ago 70m 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (54m) 4s ago 68m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (53m) 2s ago 68m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (54m) 4s ago 68m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:58:11.723 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (53m) 2s ago 68m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:58:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:11 smithi184 ceph-mon[100622]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:11 smithi184 ceph-mon[102844]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:58:12.035 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:58:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:11 smithi196 ceph-mon[76349]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:58:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:58:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:58:12.304 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:58:12.609 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:58:12.610 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:12.610 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:12.610 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:12.610 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:12 smithi196 ceph-mon[76349]: from='client.48468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:12 smithi196 ceph-mon[76349]: from='client.48474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:12 smithi196 ceph-mon[76349]: from='client.48480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1396931057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:13.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2422915368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[100622]: from='client.48468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[100622]: from='client.48474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[100622]: from='client.48480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1396931057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2422915368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[102844]: from='client.48468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[102844]: from='client.48474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[102844]: from='client.48480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1396931057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2422915368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:13.865 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:13.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:13.866 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:13.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:13.866 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:13.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:13 smithi196 ceph-mon[76349]: from='client.48492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:13 smithi196 ceph-mon[76349]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:14.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 20:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T20:58:13.812814999Z level=info msg="Completed cleanup jobs" duration=13.020688ms 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[100622]: from='client.48492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[100622]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[102844]: from='client.48492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[102844]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:15 smithi196 ceph-mon[76349]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:15 smithi184 ceph-mon[100622]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:15 smithi184 ceph-mon[102844]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:17 smithi196 ceph-mon[76349]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:17 smithi184 ceph-mon[100622]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:17 smithi184 ceph-mon[102844]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:19 smithi196 ceph-mon[76349]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:19 smithi184 ceph-mon[100622]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:19 smithi184 ceph-mon[102844]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:20.356Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:20.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:20.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:22.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:21 smithi196 ceph-mon[76349]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:21 smithi184 ceph-mon[100622]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:21 smithi184 ceph-mon[102844]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:58:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:23 smithi184 ceph-mon[100622]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:23 smithi184 ceph-mon[102844]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:23.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:23.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:23.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:23 smithi196 ceph-mon[76349]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:25 smithi196 ceph-mon[76349]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:25 smithi184 ceph-mon[100622]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:25 smithi184 ceph-mon[102844]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:27 smithi196 ceph-mon[76349]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:27 smithi184 ceph-mon[100622]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:27 smithi184 ceph-mon[102844]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:30.357Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:30.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:30.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:31 smithi184 ceph-mon[100622]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:31 smithi184 ceph-mon[102844]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:31 smithi196 ceph-mon[76349]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:58:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:33 smithi184 ceph-mon[100622]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:33 smithi184 ceph-mon[102844]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:33.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:33.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:33.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:33 smithi196 ceph-mon[76349]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:35 smithi184 ceph-mon[100622]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:35 smithi184 ceph-mon[102844]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:35 smithi196 ceph-mon[76349]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:37 smithi184 ceph-mon[100622]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:37 smithi184 ceph-mon[102844]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:37 smithi196 ceph-mon[76349]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:39.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:39 smithi196 ceph-mon[76349]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:39 smithi184 ceph-mon[100622]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:39 smithi184 ceph-mon[102844]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:40.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:40.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:40.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:41.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:41 smithi196 ceph-mon[76349]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:41 smithi184 ceph-mon[100622]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:41 smithi184 ceph-mon[102844]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:58:42.909 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:58:43.440 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:58:43.440 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 75m - - 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 75m - - 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (69m) 15s ago 70m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (70m) 14s ago 70m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (63m) 15s ago 68m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (63m) 14s ago 74m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (64m) 15s ago 76m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (62m) 15s ago 76m 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (62m) 14s ago 75m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (62m) 15s ago 74m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (70m) 15s ago 70m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (70m) 14s ago 70m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (61m) 15s ago 74m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (60m) 15s ago 73m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:58:43.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (59m) 15s ago 73m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (58m) 15s ago 73m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (57m) 14s ago 72m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (56m) 14s ago 72m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (55m) 14s ago 71m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (55m) 14s ago 71m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (63m) 14s ago 71m 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (54m) 15s ago 69m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (54m) 14s ago 69m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (54m) 15s ago 68m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:58:43.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (54m) 14s ago 68m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:58:43.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:58:43.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:58:43.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-mon[100622]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:43.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-mon[100622]: from='client.58469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:43.987 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-mon[102844]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:43.987 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-mon[102844]: from='client.58469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:43.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:43.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:43.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:43.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:43.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:43.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:44.045 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:58:44.045 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:58:44.046 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:58:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:43 smithi196 ceph-mon[76349]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:43 smithi196 ceph-mon[76349]: from='client.58469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:44.361 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:58:44.361 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:44.361 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[100622]: from='client.58475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[100622]: from='client.58481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3108127949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/860415907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:44.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[102844]: from='client.58475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:44.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[102844]: from='client.58481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:44.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:44.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3108127949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:44.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/860415907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:44 smithi196 ceph-mon[76349]: from='client.58475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:44 smithi196 ceph-mon[76349]: from='client.58481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3108127949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/860415907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:45 smithi184 ceph-mon[100622]: from='client.58493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:45 smithi184 ceph-mon[100622]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:45 smithi184 ceph-mon[102844]: from='client.58493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:45 smithi184 ceph-mon[102844]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:45 smithi196 ceph-mon[76349]: from='client.58493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:45 smithi196 ceph-mon[76349]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:47 smithi184 ceph-mon[100622]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:47 smithi184 ceph-mon[102844]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:47 smithi196 ceph-mon[76349]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:49 smithi196 ceph-mon[76349]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[100622]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[102844]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:58:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:50.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:50.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:50.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:52.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:51 smithi196 ceph-mon[76349]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:51 smithi184 ceph-mon[100622]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:51 smithi184 ceph-mon[102844]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:58:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T20:58:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:53 smithi184 ceph-mon[100622]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:53 smithi184 ceph-mon[102844]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:53.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:53.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:58:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:58:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:58:53.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:58:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:53 smithi196 ceph-mon[76349]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:55 smithi196 ceph-mon[76349]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:56.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:55 smithi184 ceph-mon[100622]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:56.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:55 smithi184 ceph-mon[102844]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:57 smithi196 ceph-mon[76349]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:57 smithi184 ceph-mon[100622]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:57 smithi184 ceph-mon[102844]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:58:59 smithi196 ceph-mon[76349]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:58:59 smithi184 ceph-mon[100622]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:58:59 smithi184 ceph-mon[102844]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:00.360Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:00.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:00.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:01 smithi196 ceph-mon[76349]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:01 smithi184 ceph-mon[100622]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:01 smithi184 ceph-mon[102844]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:59:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:03 smithi184 ceph-mon[100622]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:03 smithi184 ceph-mon[102844]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:03.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:03.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:03.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:03 smithi196 ceph-mon[76349]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:05 smithi184 ceph-mon[100622]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:05 smithi184 ceph-mon[102844]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:05 smithi196 ceph-mon[76349]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:07 smithi196 ceph-mon[76349]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:07 smithi184 ceph-mon[100622]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:07 smithi184 ceph-mon[102844]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[100622]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:08 smithi184 ceph-mon[102844]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:09.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:08 smithi196 ceph-mon[76349]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:10.361Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:10.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:10.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:11 smithi184 ceph-mon[100622]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:11 smithi184 ceph-mon[102844]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:11 smithi196 ceph-mon[76349]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T20:59:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:13 smithi184 ceph-mon[100622]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:13 smithi184 ceph-mon[102844]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:13.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:13.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:13.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:13 smithi196 ceph-mon[76349]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:14.653 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:59:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 75m - - 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 75m - - 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (70m) 7s ago 71m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (71m) 5s ago 71m 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (64m) 7s ago 69m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:59:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (64m) 5s ago 75m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (64m) 7s ago 76m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (63m) 7s ago 76m 168M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (62m) 5s ago 75m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (62m) 7s ago 75m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (70m) 7s ago 70m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (70m) 5s ago 70m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (61m) 7s ago 74m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (60m) 7s ago 74m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (59m) 7s ago 74m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (59m) 7s ago 73m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (58m) 5s ago 73m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (57m) 5s ago 72m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (56m) 5s ago 72m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (55m) 5s ago 72m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:59:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (64m) 5s ago 71m 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:59:15.182 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (55m) 7s ago 69m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:59:15.182 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (54m) 5s ago 69m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:59:15.182 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (55m) 7s ago 69m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:59:15.182 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (54m) 5s ago 69m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:59:15.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:15.492 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:59:15.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:59:15.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:59:15.492 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:59:15.763 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:59:15.764 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:59:15.764 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:59:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[102844]: from='client.58505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[102844]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[102844]: from='client.48546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2061962265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[100622]: from='client.58505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[100622]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[100622]: from='client.48546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2061962265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:15 smithi196 ceph-mon[76349]: from='client.58505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:15 smithi196 ceph-mon[76349]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:15 smithi196 ceph-mon[76349]: from='client.48546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2061962265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[102844]: from='client.58517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[102844]: from='client.48564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4163904650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[100622]: from='client.58517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[100622]: from='client.48564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4163904650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:16 smithi196 ceph-mon[76349]: from='client.58517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:16 smithi196 ceph-mon[76349]: from='client.48564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4163904650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:17 smithi184 ceph-mon[100622]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:17 smithi184 ceph-mon[102844]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:17 smithi196 ceph-mon[76349]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:19 smithi184 ceph-mon[100622]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T20:59:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:19 smithi184 ceph-mon[102844]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T20:59:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:19 smithi196 ceph-mon[76349]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T20:59:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:20.362Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:20.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:20.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:22.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:21 smithi184 ceph-mon[100622]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:21 smithi184 ceph-mon[102844]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:22.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T20:59:22.057 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:21 smithi196 ceph-mon[76349]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:23 smithi184 ceph-mon[100622]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:24.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:23 smithi184 ceph-mon[102844]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:23.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:23.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:23.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:24.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:23 smithi196 ceph-mon[76349]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:59:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:25 smithi184 ceph-mon[100622]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:25 smithi184 ceph-mon[102844]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:25 smithi196 ceph-mon[76349]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:27 smithi184 ceph-mon[100622]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:27 smithi184 ceph-mon[102844]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:27 smithi196 ceph-mon[76349]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:59:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:30.363Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:30.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:30.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:31 smithi196 ceph-mon[76349]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:31 smithi184 ceph-mon[100622]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:31 smithi184 ceph-mon[102844]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:59:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:33 smithi184 ceph-mon[100622]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:33 smithi184 ceph-mon[102844]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:33.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:33.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:34.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:33.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:33 smithi196 ceph-mon[76349]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:35 smithi184 ceph-mon[100622]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:35 smithi184 ceph-mon[102844]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:35 smithi196 ceph-mon[76349]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:37 smithi184 ceph-mon[100622]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:37 smithi184 ceph-mon[102844]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:37 smithi196 ceph-mon[76349]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:59:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:39 smithi184 ceph-mon[100622]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:39 smithi184 ceph-mon[102844]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:39 smithi196 ceph-mon[76349]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:59:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:40.364Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:40.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:40.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:41 smithi196 ceph-mon[76349]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:41 smithi184 ceph-mon[100622]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T20:59:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:41 smithi184 ceph-mon[102844]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-mon[102844]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-mon[100622]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:43.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:43.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:43.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:44.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:43 smithi196 ceph-mon[76349]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:44.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:45 smithi184 ceph-mon[100622]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:45 smithi184 ceph-mon[102844]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:45 smithi196 ceph-mon[76349]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:46.372 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 76m - - 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 76m - - 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (70m) 18s ago 71m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (71m) 17s ago 71m 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (64m) 18s ago 69m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (64m) 17s ago 75m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T20:59:46.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (65m) 18s ago 77m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (63m) 18s ago 77m 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (63m) 17s ago 76m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (63m) 18s ago 75m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (71m) 18s ago 71m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (71m) 17s ago 71m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (62m) 18s ago 75m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (61m) 18s ago 74m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (60m) 18s ago 74m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (60m) 18s ago 74m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (58m) 17s ago 73m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (57m) 17s ago 73m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (56m) 17s ago 73m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (56m) 17s ago 72m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (64m) 17s ago 72m 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T20:59:46.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (55m) 18s ago 70m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T20:59:46.948 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (55m) 17s ago 70m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T20:59:46.948 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (55m) 18s ago 69m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T20:59:46.948 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (55m) 17s ago 69m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:47.303 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T20:59:47.304 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:59:47.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T20:59:47.587 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:59:47.587 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T20:59:47.588 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T20:59:47.904 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:59:47.904 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:47.904 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:47.904 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:47.904 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:47.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[100622]: from='client.48576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:47.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[100622]: from='client.48582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[100622]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/664143319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[102844]: from='client.48576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[102844]: from='client.48582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[102844]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:47.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/664143319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:47 smithi196 ceph-mon[76349]: from='client.48576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:47 smithi196 ceph-mon[76349]: from='client.48582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:47 smithi196 ceph-mon[76349]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/664143319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='client.48588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='client.48600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/312754712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='client.48588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='client.48600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/312754712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='client.48588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='client.48600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/312754712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[100622]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[102844]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:49 smithi196 ceph-mon[76349]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T20:59:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:50.365Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:50.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:50.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:51 smithi184 ceph-mon[100622]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:52.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:51 smithi184 ceph-mon[102844]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 20:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:20:59:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T20:59:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:51 smithi196 ceph-mon[76349]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:53 smithi184 ceph-mon[100622]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:53 smithi184 ceph-mon[102844]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:53.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:53.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T20:59:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 20:59:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T20:59:53.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T20:59:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:53 smithi196 ceph-mon[76349]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:55 smithi184 ceph-mon[100622]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:55 smithi184 ceph-mon[102844]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:55 smithi196 ceph-mon[76349]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:57 smithi184 ceph-mon[100622]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:57 smithi184 ceph-mon[102844]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:57 smithi196 ceph-mon[76349]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 20:59:59 smithi184 ceph-mon[102844]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 20:59:59 smithi184 ceph-mon[100622]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T20:59:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 20:59:59 smithi196 ceph-mon[76349]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:00.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:00.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:00.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:00.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:00.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:00.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:02.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:01 smithi184 ceph-mon[100622]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:02.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:01 smithi184 ceph-mon[102844]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:02.050 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:00:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:01 smithi196 ceph-mon[76349]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:03 smithi184 ceph-mon[100622]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:03 smithi184 ceph-mon[102844]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:03.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:03.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:03.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:03 smithi196 ceph-mon[76349]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:06.079 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:05 smithi184 ceph-mon[100622]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:06.079 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:05 smithi184 ceph-mon[102844]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:05 smithi196 ceph-mon[76349]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:07.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:07 smithi184 ceph-mon[100622]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:07.965 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:07 smithi184 ceph-mon[102844]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:07 smithi196 ceph-mon[76349]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[100622]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:09.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:09 smithi184 ceph-mon[102844]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:09.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:09.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:09 smithi196 ceph-mon[76349]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:10.367Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:10.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:10.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:11 smithi196 ceph-mon[76349]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:11 smithi184 ceph-mon[100622]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:11 smithi184 ceph-mon[102844]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:00:14.108 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:13 smithi196 ceph-mon[76349]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:14.108 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-mon[102844]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-mon[100622]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:13.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:13.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:13.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:15 smithi196 ceph-mon[76349]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:15 smithi184 ceph-mon[100622]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:15 smithi184 ceph-mon[102844]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:17 smithi196 ceph-mon[76349]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:18.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:17 smithi184 ceph-mon[100622]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:18.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:17 smithi184 ceph-mon[102844]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:18.201 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 77m - - 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 76m - - 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (71m) 10s ago 72m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (72m) 8s ago 72m 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (65m) 10s ago 70m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (65m) 8s ago 76m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (65m) 10s ago 77m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (64m) 10s ago 77m 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:00:18.727 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (63m) 8s ago 76m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (64m) 10s ago 76m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (71m) 10s ago 71m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (71m) 8s ago 71m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (62m) 10s ago 75m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (61m) 10s ago 75m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (61m) 10s ago 75m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (60m) 10s ago 74m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (59m) 8s ago 74m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (58m) 8s ago 73m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (57m) 8s ago 73m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (56m) 8s ago 73m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (65m) 8s ago 72m 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (56m) 10s ago 70m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:00:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (56m) 8s ago 70m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:00:18.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (56m) 10s ago 70m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:00:18.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (55m) 8s ago 70m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:00:19.041 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:00:19.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:19.043 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:00:19.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:00:19.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:00:19.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:00:19.313 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:00:19.314 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:00:19.624 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:19.624 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:19.624 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:19.624 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:19.624 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.58577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.48618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.48624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1789745994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3324393557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.58577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.48618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.48624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1789745994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:20.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3324393557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.58577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.48618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.48624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1789745994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3324393557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:20.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:20.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:20.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:20 smithi196 ceph-mon[76349]: from='client.58601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:20 smithi184 ceph-mon[100622]: from='client.58601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:20 smithi184 ceph-mon[102844]: from='client.58601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:21 smithi196 ceph-mon[76349]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:21 smithi184 ceph-mon[100622]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:21 smithi184 ceph-mon[102844]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:00:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:00:24.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:23 smithi196 ceph-mon[76349]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:23 smithi184 ceph-mon[100622]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:23 smithi184 ceph-mon[102844]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:23.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:23.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:23.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:25 smithi196 ceph-mon[76349]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:25 smithi184 ceph-mon[102844]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:25 smithi184 ceph-mon[100622]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:27 smithi184 ceph-mon[100622]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:27 smithi184 ceph-mon[102844]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:27 smithi196 ceph-mon[76349]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:29.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:30.369Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:30.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:30.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[102844]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:30 smithi184 ceph-mon[100622]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:30 smithi196 ceph-mon[76349]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:00:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:33 smithi184 ceph-mon[100622]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:33 smithi184 ceph-mon[102844]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:33.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:33.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:33.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:33 smithi196 ceph-mon[76349]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:35 smithi184 ceph-mon[100622]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:35 smithi184 ceph-mon[102844]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:35 smithi196 ceph-mon[76349]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:37 smithi184 ceph-mon[100622]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:37 smithi184 ceph-mon[102844]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:37 smithi196 ceph-mon[76349]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:39 smithi184 ceph-mon[100622]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:39 smithi184 ceph-mon[102844]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:39 smithi196 ceph-mon[76349]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:40.370Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:40.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:40.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:41 smithi184 ceph-mon[100622]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:00:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:41 smithi184 ceph-mon[102844]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:41 smithi196 ceph-mon[76349]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:43 smithi196 ceph-mon[76349]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-mon[100622]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-mon[102844]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:43.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:43.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:43.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:45.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:45 smithi196 ceph-mon[76349]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:45 smithi184 ceph-mon[100622]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:45 smithi184 ceph-mon[102844]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:47 smithi184 ceph-mon[100622]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:47 smithi184 ceph-mon[102844]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:47 smithi196 ceph-mon[76349]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:49 smithi196 ceph-mon[76349]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[100622]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:49.931 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:49 smithi184 ceph-mon[102844]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:49.931 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 77m - - 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 77m - - 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (71m) 1s ago 72m 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (72m) 0s ago 72m 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (65m) 1s ago 70m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (65m) 0s ago 76m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (66m) 1s ago 78m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (64m) 1s ago 78m 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:00:50.486 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (64m) 0s ago 77m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (64m) 1s ago 77m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (72m) 1s ago 72m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (72m) 0s ago 72m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (63m) 1s ago 76m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (62m) 1s ago 76m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (61m) 1s ago 75m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (61m) 1s ago 75m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (59m) 0s ago 74m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (58m) 0s ago 74m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (57m) 0s ago 74m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (57m) 0s ago 73m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (65m) 0s ago 73m 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (56m) 1s ago 71m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:00:50.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (56m) 0s ago 71m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:00:50.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (56m) 1s ago 70m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:00:50.488 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (56m) 0s ago 70m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:00:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[100622]: from='client.58613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[102844]: from='client.58613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:50.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:50.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:50.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:00:50.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:00:50.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:00:50.802 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:00:50.802 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:00:50.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:00:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:00:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:50 smithi196 ceph-mon[76349]: from='client.58613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:00:51.070 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:00:51.071 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:00:51.071 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:00:51.071 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:00:51.071 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:00:51.379 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:51.379 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:51.379 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:51.379 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:51.379 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[100622]: from='client.58619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[100622]: from='client.58625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[100622]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2195992350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1248263967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[102844]: from='client.58619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[102844]: from='client.58625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[102844]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2195992350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1248263967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:51 smithi196 ceph-mon[76349]: from='client.58619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:51 smithi196 ceph-mon[76349]: from='client.58625 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:51 smithi196 ceph-mon[76349]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2195992350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1248263967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:00:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:00:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:52 smithi196 ceph-mon[76349]: from='client.48672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:52 smithi184 ceph-mon[100622]: from='client.48672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:52 smithi184 ceph-mon[102844]: from='client.48672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:53 smithi196 ceph-mon[76349]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:53 smithi184 ceph-mon[100622]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:53 smithi184 ceph-mon[102844]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:53.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:53.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:00:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:00:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:00:53.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:00:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:55 smithi184 ceph-mon[100622]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:55 smithi184 ceph-mon[102844]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:55 smithi196 ceph-mon[76349]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:57 smithi184 ceph-mon[100622]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:57 smithi184 ceph-mon[102844]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:57 smithi196 ceph-mon[76349]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:00:59 smithi184 ceph-mon[100622]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:00:59 smithi184 ceph-mon[102844]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:00:59 smithi196 ceph-mon[76349]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:00.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:00.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:00.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:02.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:01 smithi184 ceph-mon[100622]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:02.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:01 smithi184 ceph-mon[102844]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:02.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:01:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:01 smithi196 ceph-mon[76349]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:03 smithi184 ceph-mon[100622]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:03 smithi184 ceph-mon[102844]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:03.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:03.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:03.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:04.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:03 smithi196 ceph-mon[76349]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:05 smithi184 ceph-mon[100622]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:05 smithi184 ceph-mon[102844]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:05 smithi196 ceph-mon[76349]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:07 smithi184 ceph-mon[100622]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:07 smithi184 ceph-mon[102844]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:07 smithi196 ceph-mon[76349]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[100622]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:09.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[102844]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:09.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:09 smithi196 ceph-mon[76349]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:10.617 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:10.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:10.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:10.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:10.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:10.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[102844]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:11.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:11 smithi184 ceph-mon[100622]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:11 smithi196 ceph-mon[76349]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:01:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-mon[100622]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-mon[102844]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:13.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:13.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:13.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:13 smithi196 ceph-mon[76349]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:15 smithi184 ceph-mon[100622]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:15 smithi184 ceph-mon[102844]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:15 smithi196 ceph-mon[76349]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:17 smithi184 ceph-mon[100622]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:17 smithi184 ceph-mon[102844]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:17 smithi196 ceph-mon[76349]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:19 smithi184 ceph-mon[100622]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:19 smithi184 ceph-mon[102844]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:19 smithi196 ceph-mon[76349]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:20.373Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:20.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:20.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:21.679 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:01:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:21 smithi184 ceph-mon[100622]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:21 smithi184 ceph-mon[102844]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:21 smithi196 ceph-mon[76349]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:22.209 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:01:22.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 78m - - 2024-09-20T21:01:22.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 77m - - 2024-09-20T21:01:22.209 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (72m) 13s ago 73m 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:01:22.209 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (73m) 11s ago 73m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (66m) 13s ago 71m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (66m) 11s ago 77m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (67m) 13s ago 78m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (65m) 13s ago 79m 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (64m) 11s ago 77m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (65m) 13s ago 77m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (72m) 13s ago 72m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (72m) 11s ago 72m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (63m) 13s ago 76m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (62m) 13s ago 76m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (62m) 13s ago 76m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (61m) 13s ago 75m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (60m) 11s ago 75m 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:01:22.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (59m) 11s ago 74m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (58m) 11s ago 74m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (57m) 11s ago 74m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (66m) 11s ago 73m 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (57m) 13s ago 71m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (57m) 11s ago 71m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (57m) 13s ago 71m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:01:22.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (56m) 11s ago 71m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:01:22.518 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:01:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:01:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:01:22.520 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:01:22.788 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:01:22.788 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:01:22.788 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:01:22.788 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:01:22.789 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:22 smithi184 ceph-mon[102844]: from='client.48684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2863011610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:22 smithi184 ceph-mon[100622]: from='client.48684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2863011610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:23.100 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:01:23.100 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:23.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:23.100 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:23.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:22 smithi196 ceph-mon[76349]: from='client.48684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2863011610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[100622]: from='client.48690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[100622]: from='client.58661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[100622]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[100622]: from='client.48708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1783416006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[102844]: from='client.48690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[102844]: from='client.58661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[102844]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[102844]: from='client.48708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1783416006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:23.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:23.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:24.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:23.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:23 smithi196 ceph-mon[76349]: from='client.48690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:23 smithi196 ceph-mon[76349]: from='client.58661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:23 smithi196 ceph-mon[76349]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:23 smithi196 ceph-mon[76349]: from='client.48708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1783416006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:26.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:25 smithi196 ceph-mon[76349]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:25 smithi184 ceph-mon[100622]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:25 smithi184 ceph-mon[102844]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:27 smithi196 ceph-mon[76349]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:27 smithi184 ceph-mon[100622]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:27 smithi184 ceph-mon[102844]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:28.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:28.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[100622]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[102844]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:29 smithi196 ceph-mon[76349]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:30.374Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:30.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:30.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[102844]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:31 smithi184 ceph-mon[100622]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:31 smithi196 ceph-mon[76349]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:01:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:33 smithi184 ceph-mon[100622]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:33 smithi184 ceph-mon[102844]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:33.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:33.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:33.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:33 smithi196 ceph-mon[76349]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:35 smithi184 ceph-mon[100622]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:35 smithi184 ceph-mon[102844]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:35 smithi196 ceph-mon[76349]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:37 smithi184 ceph-mon[100622]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:37 smithi184 ceph-mon[102844]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:37 smithi196 ceph-mon[76349]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:39 smithi184 ceph-mon[100622]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:39 smithi184 ceph-mon[102844]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:39 smithi196 ceph-mon[76349]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:40.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:40.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:40.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:41 smithi196 ceph-mon[76349]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:42.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:41 smithi184 ceph-mon[100622]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:42.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:41 smithi184 ceph-mon[102844]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:42.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:01:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:43 smithi196 ceph-mon[76349]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-mon[102844]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-mon[100622]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:43.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:43.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:43.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:45 smithi196 ceph-mon[76349]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:45 smithi184 ceph-mon[100622]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:45 smithi184 ceph-mon[102844]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:47.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:47 smithi196 ceph-mon[76349]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:47 smithi184 ceph-mon[100622]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:47 smithi184 ceph-mon[102844]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:49.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:49.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[102844]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[100622]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:49.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:49 smithi196 ceph-mon[76349]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:50.605 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:50.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:50.605 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:50.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:50.605 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:50.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[100622]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:51 smithi184 ceph-mon[102844]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:51 smithi196 ceph-mon[76349]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:01:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:01:53.397 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 78m - - 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 78m - - 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (72m) 4s ago 73m 28.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (73m) 3s ago 73m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (66m) 4s ago 71m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (66m) 3s ago 77m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (67m) 4s ago 79m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (65m) 4s ago 79m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (65m) 3s ago 78m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (65m) 4s ago 78m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (73m) 4s ago 73m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:01:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (73m) 3s ago 73m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (64m) 4s ago 77m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (63m) 4s ago 77m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (62m) 4s ago 76m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (62m) 4s ago 76m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (60m) 3s ago 75m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (59m) 3s ago 75m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (58m) 3s ago 75m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (58m) 3s ago 74m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (67m) 3s ago 74m 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (57m) 4s ago 72m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (57m) 3s ago 72m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (57m) 4s ago 72m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:01:53.927 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (57m) 3s ago 71m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:01:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:53 smithi184 ceph-mon[102844]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:53 smithi184 ceph-mon[100622]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:53.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:53.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:01:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:01:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:01:53.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:01:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:53 smithi196 ceph-mon[76349]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:54.238 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:01:54.238 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:01:54.238 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:01:54.239 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:01:54.506 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:01:54.506 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:01:54.506 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:01:54.506 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:01:54.507 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:01:54.822 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:01:54.822 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:54.822 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:54.822 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:54.822 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:54 smithi196 ceph-mon[76349]: from='client.58685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:54 smithi196 ceph-mon[76349]: from='client.48726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:54 smithi196 ceph-mon[76349]: from='client.58697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/43578423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[102844]: from='client.58685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[102844]: from='client.48726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[102844]: from='client.58697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/43578423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[100622]: from='client.58685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[100622]: from='client.48726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[100622]: from='client.58697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/43578423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:56.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:55 smithi196 ceph-mon[76349]: from='client.48744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:56.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:55 smithi196 ceph-mon[76349]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:56.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2968778148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[102844]: from='client.48744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[102844]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2968778148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[100622]: from='client.48744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[100622]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2968778148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:57 smithi196 ceph-mon[76349]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:57 smithi184 ceph-mon[100622]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:57 smithi184 ceph-mon[102844]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:01:59 smithi196 ceph-mon[76349]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:01:59 smithi184 ceph-mon[100622]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:01:59 smithi184 ceph-mon[102844]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:00.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:00.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:00.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:01 smithi196 ceph-mon[76349]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:01 smithi184 ceph-mon[100622]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:01 smithi184 ceph-mon[102844]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T21:02:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:03 smithi196 ceph-mon[76349]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:03 smithi184 ceph-mon[102844]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:03 smithi184 ceph-mon[100622]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:03.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:03.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:03.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:06.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:05 smithi196 ceph-mon[76349]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:05 smithi184 ceph-mon[102844]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:05 smithi184 ceph-mon[100622]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:07 smithi196 ceph-mon[76349]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:07 smithi184 ceph-mon[102844]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:07 smithi184 ceph-mon[100622]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[102844]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[100622]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:09 smithi196 ceph-mon[76349]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:10.377Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:10.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:10.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[100622]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:11 smithi184 ceph-mon[102844]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:11.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:11 smithi196 ceph-mon[76349]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T21:02:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:13 smithi196 ceph-mon[76349]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-mon[100622]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-mon[102844]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:13.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:13.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:13.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:15 smithi196 ceph-mon[76349]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:15 smithi184 ceph-mon[100622]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:15 smithi184 ceph-mon[102844]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:17 smithi196 ceph-mon[76349]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:17 smithi184 ceph-mon[100622]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:17 smithi184 ceph-mon[102844]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:19 smithi196 ceph-mon[76349]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:19 smithi184 ceph-mon[100622]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:19 smithi184 ceph-mon[102844]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:20.378Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:20.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:20.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:21 smithi196 ceph-mon[76349]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:21 smithi184 ceph-mon[100622]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:02:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:21 smithi184 ceph-mon[102844]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:23 smithi196 ceph-mon[76349]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:23 smithi184 ceph-mon[100622]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:23 smithi184 ceph-mon[102844]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:23.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:23.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:23.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:25.119 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 79m - - 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 78m - - 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (73m) 16s ago 74m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (74m) 15s ago 74m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (67m) 16s ago 72m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (67m) 15s ago 78m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (68m) 16s ago 80m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (66m) 16s ago 80m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (65m) 15s ago 78m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (66m) 16s ago 78m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (74m) 16s ago 73m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:02:25.647 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (73m) 15s ago 73m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (64m) 16s ago 77m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (63m) 16s ago 77m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (63m) 16s ago 77m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (62m) 16s ago 76m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (61m) 15s ago 76m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (60m) 15s ago 76m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (59m) 15s ago 75m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (58m) 15s ago 75m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (67m) 15s ago 74m 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (58m) 16s ago 72m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (58m) 15s ago 73m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (58m) 16s ago 72m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:02:25.648 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (58m) 15s ago 72m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:02:25.960 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:02:25.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:25 smithi184 ceph-mon[102844]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:25.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:25 smithi184 ceph-mon[100622]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:25 smithi196 ceph-mon[76349]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:02:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:02:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:02:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:02:26.229 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:02:26.537 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:02:26.537 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:26.537 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:26.537 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:26.537 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:27.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:26 smithi196 ceph-mon[76349]: from='client.48756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:26 smithi196 ceph-mon[76349]: from='client.48762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:26 smithi196 ceph-mon[76349]: from='client.58733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2310269911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:27.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2449619413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[100622]: from='client.48756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[100622]: from='client.48762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[100622]: from='client.58733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2310269911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2449619413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[102844]: from='client.48756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[102844]: from='client.48762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[102844]: from='client.58733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2310269911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2449619413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:28.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:27 smithi196 ceph-mon[76349]: from='client.58745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:28.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:27 smithi196 ceph-mon[76349]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:27 smithi184 ceph-mon[102844]: from='client.58745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:27 smithi184 ceph-mon[102844]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:27 smithi184 ceph-mon[100622]: from='client.58745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:27 smithi184 ceph-mon[100622]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:29.057 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:29 smithi196 ceph-mon[76349]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[100622]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[102844]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:30.652 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:30.378Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:30.652 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:30.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:30.652 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:30.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[100622]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:31 smithi184 ceph-mon[102844]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:31 smithi196 ceph-mon[76349]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:32.268 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:02:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:33 smithi196 ceph-mon[76349]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:33 smithi184 ceph-mon[102844]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:33 smithi184 ceph-mon[100622]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:33.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:33.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:33.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:35 smithi196 ceph-mon[76349]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:35 smithi184 ceph-mon[102844]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:35 smithi184 ceph-mon[100622]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:37 smithi196 ceph-mon[76349]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:37 smithi184 ceph-mon[102844]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:37 smithi184 ceph-mon[100622]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:39 smithi196 ceph-mon[76349]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:39 smithi184 ceph-mon[100622]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:39 smithi184 ceph-mon[102844]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:40.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:40.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:40.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:41 smithi196 ceph-mon[76349]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:41 smithi184 ceph-mon[100622]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:41 smithi184 ceph-mon[102844]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:02:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:43 smithi196 ceph-mon[76349]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-mon[102844]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-mon[100622]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:43.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:43.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:43.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:45 smithi196 ceph-mon[76349]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:45 smithi184 ceph-mon[100622]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:45 smithi184 ceph-mon[102844]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:47.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:47 smithi196 ceph-mon[76349]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:47 smithi184 ceph-mon[100622]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:47 smithi184 ceph-mon[102844]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:49 smithi196 ceph-mon[76349]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[100622]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[102844]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:50.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:50.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:50.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:51 smithi196 ceph-mon[76349]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.186 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:02:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:02:52.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[100622]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:52.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[102844]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:02:54.102 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:53 smithi184 ceph-mon[100622]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:54.103 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:53 smithi184 ceph-mon[102844]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:54.103 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:53.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:54.103 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:53.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:02:54.103 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:02:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:02:53.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:02:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:53 smithi196 ceph-mon[76349]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:55 smithi196 ceph-mon[76349]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:55 smithi184 ceph-mon[100622]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:55 smithi184 ceph-mon[102844]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:56.833 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:02:57.359 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:02:57.359 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 79m - - 2024-09-20T21:02:57.359 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 79m - - 2024-09-20T21:02:57.359 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (73m) 7s ago 74m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:02:57.359 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (74m) 6s ago 74m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (67m) 7s ago 72m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (67m) 6s ago 78m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (68m) 7s ago 80m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (66m) 7s ago 80m 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (66m) 6s ago 79m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (66m) 7s ago 79m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (74m) 7s ago 74m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (74m) 6s ago 74m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (65m) 7s ago 78m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (64m) 7s ago 78m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (63m) 7s ago 77m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (63m) 7s ago 77m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (61m) 6s ago 76m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:02:57.360 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (61m) 6s ago 76m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (59m) 6s ago 76m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (59m) 6s ago 75m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (68m) 6s ago 75m 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (58m) 7s ago 73m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (58m) 6s ago 73m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (58m) 7s ago 73m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:02:57.361 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (58m) 6s ago 73m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:02:57.670 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:02:57.671 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:02:57.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[100622]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:57.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[100622]: from='client.58757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:57.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/433813411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:57.941 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:02:57.941 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:02:57.941 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:02:57.941 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:02:57.941 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:02:57.942 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:02:57.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:02:57.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:02:57.942 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:02:57.942 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:02:57.943 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:02:57.943 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:02:57.943 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:02:57.943 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:02:57.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[102844]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:57.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[102844]: from='client.58757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:57.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/433813411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:57 smithi196 ceph-mon[76349]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:57 smithi196 ceph-mon[76349]: from='client.58757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/433813411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:02:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:58.253 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:58.253 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[100622]: from='client.48798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[100622]: from='client.48804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[100622]: from='client.48816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2300046771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[102844]: from='client.48798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[102844]: from='client.48804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[102844]: from='client.48816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2300046771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:58 smithi196 ceph-mon[76349]: from='client.48798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:58 smithi196 ceph-mon[76349]: from='client.48804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:58 smithi196 ceph-mon[76349]: from='client.48816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2300046771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:02:59 smithi196 ceph-mon[76349]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:02:59 smithi184 ceph-mon[100622]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:02:59 smithi184 ceph-mon[102844]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:00.380Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:00.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:00.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:01 smithi196 ceph-mon[76349]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:01 smithi184 ceph-mon[100622]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:03:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:01 smithi184 ceph-mon[102844]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:03 smithi196 ceph-mon[76349]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:03 smithi184 ceph-mon[100622]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:03 smithi184 ceph-mon[102844]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:03.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:03.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:03.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:05 smithi196 ceph-mon[76349]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:05 smithi184 ceph-mon[100622]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:05 smithi184 ceph-mon[102844]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:07 smithi196 ceph-mon[76349]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:07 smithi184 ceph-mon[100622]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:07 smithi184 ceph-mon[102844]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:09.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:09.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:09 smithi196 ceph-mon[76349]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:09.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:09.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:09.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[100622]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[102844]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:10.381Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:10.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:10.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:12.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[100622]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:12.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[102844]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:11 smithi196 ceph-mon[76349]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:14.079 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-mon[100622]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:14.079 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:14.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-mon[102844]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:14.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:14.080 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:13.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:14.080 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:13.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:14.080 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:13.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:13 smithi196 ceph-mon[76349]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:15 smithi184 ceph-mon[100622]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:15 smithi184 ceph-mon[102844]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:15 smithi196 ceph-mon[76349]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:17 smithi184 ceph-mon[100622]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:17 smithi184 ceph-mon[102844]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:17 smithi196 ceph-mon[76349]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:19 smithi184 ceph-mon[100622]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:19 smithi184 ceph-mon[102844]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:19 smithi196 ceph-mon[76349]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:20.382Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:20.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:20.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:21 smithi184 ceph-mon[100622]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:21 smithi184 ceph-mon[102844]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:03:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:21 smithi196 ceph-mon[76349]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:23 smithi184 ceph-mon[100622]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:23 smithi184 ceph-mon[102844]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:23.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:23.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:23.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:23 smithi196 ceph-mon[76349]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:25 smithi184 ceph-mon[100622]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:25 smithi184 ceph-mon[102844]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:25 smithi196 ceph-mon[76349]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:28.193 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:27 smithi196 ceph-mon[76349]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:27 smithi184 ceph-mon[100622]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:27 smithi184 ceph-mon[102844]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:28.549 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:03:29.086 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:03:29.086 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 80m - - 2024-09-20T21:03:29.086 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 79m - - 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (74m) 19s ago 75m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (75m) 18s ago 75m 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (68m) 19s ago 73m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (68m) 18s ago 79m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (69m) 19s ago 81m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (67m) 19s ago 81m 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (66m) 18s ago 79m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (67m) 19s ago 79m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (75m) 19s ago 75m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (74m) 18s ago 74m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (65m) 19s ago 79m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (64m) 19s ago 78m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (64m) 19s ago 78m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:03:29.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (63m) 19s ago 77m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (62m) 18s ago 77m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (61m) 18s ago 77m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (60m) 18s ago 76m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (59m) 18s ago 76m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (68m) 18s ago 75m 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (59m) 19s ago 73m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (59m) 18s ago 74m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (59m) 19s ago 73m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:03:29.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (59m) 18s ago 73m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:03:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:03:29.396 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:03:29.679 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:03:29.680 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:03:29.992 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:03:29.992 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:29.992 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:29.992 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:29.992 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:29.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='client.58793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:29.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:29.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='client.48834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1880082607' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='client.58793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='client.48834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1880082607' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:29.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='client.58793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='client.48834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1880082607' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:30.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:30.382Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:30.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:30.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:30 smithi196 ceph-mon[76349]: from='client.58805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:30 smithi196 ceph-mon[76349]: from='client.58817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4012517919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[100622]: from='client.58805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[100622]: from='client.58817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4012517919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[102844]: from='client.58805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[102844]: from='client.58817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4012517919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[100622]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[102844]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:03:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:31 smithi196 ceph-mon[76349]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:33 smithi184 ceph-mon[100622]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:33 smithi184 ceph-mon[102844]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:33.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:33.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:33.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:33 smithi196 ceph-mon[76349]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:35 smithi184 ceph-mon[100622]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:35 smithi184 ceph-mon[102844]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:35 smithi196 ceph-mon[76349]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:37 smithi184 ceph-mon[100622]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:37 smithi184 ceph-mon[102844]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:37 smithi196 ceph-mon[76349]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:39 smithi184 ceph-mon[100622]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:39 smithi184 ceph-mon[102844]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:39 smithi196 ceph-mon[76349]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:40.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:40.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:40.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:41 smithi184 ceph-mon[100622]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:03:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:41 smithi184 ceph-mon[102844]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:41 smithi196 ceph-mon[76349]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:43.945 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:43.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:43.945 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:43.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:43.946 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:43.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-mon[100622]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-mon[102844]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:43 smithi196 ceph-mon[76349]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:45 smithi184 ceph-mon[100622]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:45 smithi184 ceph-mon[102844]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:45 smithi196 ceph-mon[76349]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:47 smithi184 ceph-mon[100622]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:47 smithi184 ceph-mon[102844]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:47 smithi196 ceph-mon[76349]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[100622]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[102844]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:50.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:49 smithi196 ceph-mon[76349]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:50.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:50.384Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:50.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:50.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:03:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[102844]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[100622]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:51 smithi196 ceph-mon[76349]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:52.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:03:54.076 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:53 smithi184 ceph-mon[100622]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:54.077 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:53 smithi184 ceph-mon[102844]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:54.077 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:53.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:54.077 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:53.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:03:54.077 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:03:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:03:53.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:03:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:53 smithi196 ceph-mon[76349]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:55 smithi184 ceph-mon[100622]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:55 smithi184 ceph-mon[102844]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:55 smithi196 ceph-mon[76349]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:58.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:57 smithi196 ceph-mon[76349]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:57 smithi184 ceph-mon[102844]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:57 smithi184 ceph-mon[100622]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:03:59 smithi196 ceph-mon[76349]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:00.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:00.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:03:59 smithi184 ceph-mon[100622]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:00.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:00.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:03:59 smithi184 ceph-mon[102844]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:00.288 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:04:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:00.384Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:00.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:00.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:00.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 80m - - 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 80m - - 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (74m) 11s ago 76m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (75m) 9s ago 75m 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (68m) 11s ago 73m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:04:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (68m) 9s ago 79m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (69m) 11s ago 81m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (68m) 11s ago 81m 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (67m) 9s ago 80m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (67m) 11s ago 80m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (75m) 11s ago 75m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (75m) 9s ago 75m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (66m) 11s ago 79m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (65m) 11s ago 79m 68.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (64m) 11s ago 78m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (64m) 11s ago 78m 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (62m) 9s ago 78m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (62m) 9s ago 77m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (60m) 9s ago 77m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (60m) 9s ago 76m 71.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:04:00.818 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (69m) 9s ago 76m 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:04:00.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (59m) 11s ago 74m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:04:00.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (59m) 9s ago 74m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:04:00.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (59m) 11s ago 74m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:04:00.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (59m) 9s ago 74m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:04:01.130 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:04:01.130 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:04:01.130 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:04:01.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:04:01.132 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:04:01.399 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:04:01.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:04:01.714 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:04:01.715 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:01.715 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:01.715 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: from='client.58829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: from='client.58835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: from='client.58841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/53679330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1504573398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: from='client.58829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: from='client.58835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: from='client.58841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/53679330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1504573398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: from='client.58829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: from='client.58835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: from='client.58841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/53679330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1504573398' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:04:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:02 smithi196 ceph-mon[76349]: from='client.58853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:03.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:02 smithi184 ceph-mon[100622]: from='client.58853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:03.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:02 smithi184 ceph-mon[102844]: from='client.58853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:03 smithi196 ceph-mon[76349]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:03 smithi184 ceph-mon[100622]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:03 smithi184 ceph-mon[102844]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:03.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:03.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:03.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:05 smithi196 ceph-mon[76349]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:05 smithi184 ceph-mon[100622]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:05 smithi184 ceph-mon[102844]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:07 smithi196 ceph-mon[76349]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:04:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:07 smithi184 ceph-mon[100622]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:04:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:07 smithi184 ceph-mon[102844]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:04:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:09 smithi196 ceph-mon[76349]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:04:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:09 smithi184 ceph-mon[100622]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:04:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:09 smithi184 ceph-mon[102844]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:04:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:10.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:10.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:10.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:11 smithi196 ceph-mon[76349]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:04:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[100622]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[102844]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-mon[100622]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:04:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-mon[102844]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:04:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:13.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:13.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:13.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:13 smithi196 ceph-mon[76349]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:04:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:15 smithi184 ceph-mon[100622]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T21:04:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:15 smithi184 ceph-mon[102844]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T21:04:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:15 smithi196 ceph-mon[76349]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T21:04:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:17 smithi184 ceph-mon[100622]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:17 smithi184 ceph-mon[102844]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:17 smithi196 ceph-mon[76349]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:19 smithi184 ceph-mon[102844]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:19 smithi184 ceph-mon[100622]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:19 smithi196 ceph-mon[76349]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:20.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:20.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:20.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:21 smithi184 ceph-mon[100622]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:04:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:21 smithi184 ceph-mon[102844]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:04:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:04:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:21 smithi196 ceph-mon[76349]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:04:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:23 smithi184 ceph-mon[100622]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:23 smithi184 ceph-mon[102844]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:23.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:23.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:23.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:23 smithi196 ceph-mon[76349]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T21:04:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:25 smithi184 ceph-mon[100622]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:25 smithi184 ceph-mon[102844]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:25 smithi196 ceph-mon[76349]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:28.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:27 smithi196 ceph-mon[76349]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:27 smithi184 ceph-mon[100622]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:27 smithi184 ceph-mon[102844]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T21:04:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:29.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:30.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:30.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:29 smithi196 ceph-mon[76349]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:29 smithi184 ceph-mon[100622]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:29 smithi184 ceph-mon[102844]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T21:04:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:30.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:30.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:30.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:31.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:31 smithi196 ceph-mon[76349]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:04:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[100622]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:31 smithi184 ceph-mon[102844]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:04:32.015 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:04:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 81m - - 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 81m - - 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (75m) 2s ago 76m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (76m) 1s ago 76m 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (69m) 2s ago 74m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (69m) 1s ago 80m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (70m) 2s ago 82m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (68m) 2s ago 82m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (67m) 1s ago 80m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (68m) 2s ago 80m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (76m) 2s ago 76m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:04:32.561 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (76m) 1s ago 76m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (66m) 2s ago 80m 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (65m) 2s ago 79m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (65m) 2s ago 79m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (64m) 2s ago 78m 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (63m) 1s ago 78m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (62m) 1s ago 78m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (61m) 1s ago 77m 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (60m) 1s ago 77m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (69m) 1s ago 76m 67.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (60m) 2s ago 75m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (60m) 1s ago 75m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (60m) 2s ago 74m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:04:32.562 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (60m) 1s ago 74m 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:04:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:04:32.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:04:32.897 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:04:33.174 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:04:33.175 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:04:33.175 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:04:33.175 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:04:33.175 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:04:33.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:04:33.508 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:04:33.509 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:33.509 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:33.509 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:33.509 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: from='client.48900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: from='client.58871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: from='client.58877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4225418590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/490901977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: from='client.48900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: from='client.58871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: from='client.58877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:04:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4225418590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/490901977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: from='client.48900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: from='client.58871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: from='client.58877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:04:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4225418590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/490901977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:34.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:33.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:34.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:33.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:34.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:33.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:34 smithi184 ceph-mon[100622]: from='client.48924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:34 smithi184 ceph-mon[102844]: from='client.48924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:34 smithi196 ceph-mon[76349]: from='client.48924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:35 smithi184 ceph-mon[100622]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:04:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:35 smithi184 ceph-mon[102844]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:35 smithi196 ceph-mon[76349]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:04:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:37 smithi196 ceph-mon[76349]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-20T21:04:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:37 smithi184 ceph-mon[100622]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-20T21:04:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:37 smithi184 ceph-mon[102844]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-20T21:04:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:39 smithi196 ceph-mon[76349]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:04:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:39 smithi184 ceph-mon[100622]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:04:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:39 smithi184 ceph-mon[102844]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:04:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:40.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:40.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:40.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:41 smithi196 ceph-mon[76349]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:41 smithi184 ceph-mon[100622]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:41 smithi184 ceph-mon[102844]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:04:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:04:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:43 smithi196 ceph-mon[76349]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-mon[100622]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-mon[102844]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:43.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:43.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:43.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:45 smithi196 ceph-mon[76349]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:45 smithi184 ceph-mon[102844]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:45 smithi184 ceph-mon[100622]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:47 smithi196 ceph-mon[76349]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:47 smithi184 ceph-mon[100622]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:47 smithi184 ceph-mon[102844]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:49 smithi196 ceph-mon[76349]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:50.143 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:50.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:49 smithi184 ceph-mon[100622]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:50.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:50.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:49 smithi184 ceph-mon[102844]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:50.387Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:50.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:50.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:51 smithi196 ceph-mon[76349]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[100622]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:51 smithi184 ceph-mon[102844]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:04:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:04:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:04:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:53 smithi184 ceph-mon[100622]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:53 smithi184 ceph-mon[102844]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:53.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:53.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:04:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:04:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:04:53.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:04:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:53 smithi196 ceph-mon[76349]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:55 smithi196 ceph-mon[76349]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:56.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:55 smithi184 ceph-mon[100622]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:56.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:55 smithi184 ceph-mon[102844]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:57 smithi184 ceph-mon[100622]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:57 smithi184 ceph-mon[102844]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:57 smithi196 ceph-mon[76349]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:04:59 smithi196 ceph-mon[76349]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:04:59 smithi184 ceph-mon[100622]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:04:59 smithi184 ceph-mon[102844]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:00.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:00.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:00.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:01 smithi196 ceph-mon[76349]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:01 smithi184 ceph-mon[100622]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:01 smithi184 ceph-mon[102844]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:05:03.807 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:05:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:03 smithi196 ceph-mon[76349]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:03 smithi184 ceph-mon[100622]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:03 smithi184 ceph-mon[102844]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:03.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:03.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:03.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:04.337 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:05:04.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 81m - - 2024-09-20T21:05:04.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 81m - - 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (75m) 14s ago 77m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (76m) 12s ago 76m 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (69m) 14s ago 75m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (69m) 12s ago 80m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (70m) 14s ago 82m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (69m) 14s ago 82m 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (68m) 12s ago 81m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (68m) 14s ago 81m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (76m) 14s ago 76m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (76m) 12s ago 76m 16.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (67m) 14s ago 80m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (66m) 14s ago 80m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:05:04.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (65m) 14s ago 79m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (65m) 14s ago 79m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (63m) 12s ago 79m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (63m) 12s ago 78m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (62m) 12s ago 78m 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (61m) 12s ago 77m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (70m) 12s ago 77m 67.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (61m) 14s ago 75m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (60m) 12s ago 75m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (60m) 14s ago 75m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:05:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (60m) 12s ago 75m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:04.649 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:05:04.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:05:04.922 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:05:04.923 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:05:04.923 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:05:04.923 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:05:04.923 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:05:04.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:04 smithi184 ceph-mon[100622]: from='client.58901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:04.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2939308511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:04.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:04 smithi184 ceph-mon[102844]: from='client.58901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:04.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2939308511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:05.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:04 smithi196 ceph-mon[76349]: from='client.58901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:05.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2939308511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:05.232 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:05:05.232 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:05.232 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:05.232 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:05.232 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:05 smithi196 ceph-mon[76349]: from='client.48942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:05 smithi196 ceph-mon[76349]: from='client.48948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:05 smithi196 ceph-mon[76349]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:05 smithi196 ceph-mon[76349]: from='client.58925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1444349199' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[100622]: from='client.48942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[100622]: from='client.48948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[100622]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[100622]: from='client.58925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1444349199' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[102844]: from='client.48942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[102844]: from='client.48948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[102844]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[102844]: from='client.58925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1444349199' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:07 smithi196 ceph-mon[76349]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:07 smithi184 ceph-mon[100622]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:07 smithi184 ceph-mon[102844]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:09 smithi196 ceph-mon[76349]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:10.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:10.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:09 smithi184 ceph-mon[100622]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:10.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:10.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:09 smithi184 ceph-mon[102844]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:10.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:10.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:10.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:10.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:10.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[100622]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:11 smithi184 ceph-mon[102844]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:11.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:11.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:11 smithi196 ceph-mon[76349]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:13 smithi196 ceph-mon[76349]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-mon[100622]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-mon[102844]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:13.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:13.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:13.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:15 smithi196 ceph-mon[76349]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:15 smithi184 ceph-mon[100622]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:15 smithi184 ceph-mon[102844]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:17 smithi184 ceph-mon[100622]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:17 smithi184 ceph-mon[102844]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:17 smithi196 ceph-mon[76349]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:19 smithi196 ceph-mon[76349]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:20.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:19 smithi184 ceph-mon[100622]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:19 smithi184 ceph-mon[102844]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:20.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:20.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:20.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:21 smithi196 ceph-mon[76349]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:21 smithi184 ceph-mon[100622]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:21 smithi184 ceph-mon[102844]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:05:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:23 smithi196 ceph-mon[76349]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:23 smithi184 ceph-mon[100622]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:23 smithi184 ceph-mon[102844]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:23.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:23.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:23.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:25 smithi196 ceph-mon[76349]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:25 smithi184 ceph-mon[100622]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:25 smithi184 ceph-mon[102844]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:27 smithi196 ceph-mon[76349]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:27 smithi184 ceph-mon[100622]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:27 smithi184 ceph-mon[102844]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:29 smithi196 ceph-mon[76349]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:29 smithi184 ceph-mon[102844]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:29 smithi184 ceph-mon[100622]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:30.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:30.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:30.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[100622]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:31 smithi184 ceph-mon[102844]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:31 smithi196 ceph-mon[76349]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:05:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:33.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:33 smithi196 ceph-mon[76349]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:33 smithi184 ceph-mon[100622]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:33 smithi184 ceph-mon[102844]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:33.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:33.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:33.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:35.525 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 82m - - 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 82m - - 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (76m) 5s ago 77m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (77m) 4s ago 77m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (70m) 5s ago 75m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:05:36.056 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (70m) 4s ago 81m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (71m) 5s ago 83m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (69m) 5s ago 83m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (69m) 4s ago 81m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (69m) 5s ago 81m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (77m) 5s ago 77m 16.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (77m) 4s ago 77m 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (67m) 5s ago 81m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (66m) 5s ago 80m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (66m) 5s ago 80m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (65m) 5s ago 79m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (64m) 4s ago 79m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (63m) 4s ago 79m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (62m) 4s ago 78m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:05:36.057 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (61m) 4s ago 78m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:05:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (70m) 4s ago 77m 68.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:05:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (61m) 5s ago 76m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:05:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (61m) 4s ago 76m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:05:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (61m) 5s ago 75m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:05:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (61m) 4s ago 75m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:05:36.282 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:35 smithi184 ceph-mon[100622]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:36.283 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:35 smithi184 ceph-mon[102844]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:36.381 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:05:36.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:05:36.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:05:36.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:05:36.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:05:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:35 smithi196 ceph-mon[76349]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:05:36.661 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:05:36.662 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:05:36.981 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:05:36.982 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:36.982 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[100622]: from='client.48972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[100622]: from='client.48978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1910700561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[102844]: from='client.48972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[102844]: from='client.48978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1910700561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:36 smithi196 ceph-mon[76349]: from='client.48972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:36 smithi196 ceph-mon[76349]: from='client.48978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1910700561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[100622]: from='client.48984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[100622]: from='client.48996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[100622]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/86365286' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[102844]: from='client.48984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[102844]: from='client.48996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[102844]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:38.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/86365286' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:37 smithi196 ceph-mon[76349]: from='client.48984 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:37 smithi196 ceph-mon[76349]: from='client.48996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:37 smithi196 ceph-mon[76349]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/86365286' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:39 smithi184 ceph-mon[100622]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:39 smithi184 ceph-mon[102844]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:39 smithi196 ceph-mon[76349]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:40.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:40.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:40.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:41 smithi184 ceph-mon[100622]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:41 smithi184 ceph-mon[102844]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:05:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:41 smithi196 ceph-mon[76349]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-mon[100622]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-mon[102844]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:43.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:43.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:43.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:43 smithi196 ceph-mon[76349]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:45 smithi184 ceph-mon[100622]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:45 smithi184 ceph-mon[102844]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:45 smithi196 ceph-mon[76349]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:48.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:47 smithi184 ceph-mon[100622]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:48.256 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:47 smithi184 ceph-mon[102844]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:47 smithi196 ceph-mon[76349]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:49 smithi184 ceph-mon[100622]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:49 smithi184 ceph-mon[102844]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:49 smithi196 ceph-mon[76349]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:50.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:50.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:50.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:50.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[100622]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:51 smithi184 ceph-mon[102844]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:51 smithi196 ceph-mon[76349]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:05:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[100622]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-mon[102844]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:05:53.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:53 smithi196 ceph-mon[76349]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:53.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:53.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:05:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:05:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:05:53.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:05:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:55 smithi196 ceph-mon[76349]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:55 smithi184 ceph-mon[100622]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:55 smithi184 ceph-mon[102844]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:57 smithi196 ceph-mon[76349]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:57 smithi184 ceph-mon[100622]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:57 smithi184 ceph-mon[102844]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:59.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:59.100 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:59 smithi196 ceph-mon[76349]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:05:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:59 smithi184 ceph-mon[102844]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:05:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:59 smithi184 ceph-mon[100622]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:05:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:00.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:00.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:00.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:01 smithi196 ceph-mon[76349]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:01 smithi184 ceph-mon[100622]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:01 smithi184 ceph-mon[102844]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:06:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:03 smithi196 ceph-mon[76349]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:03 smithi184 ceph-mon[100622]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:03 smithi184 ceph-mon[102844]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:03.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:03.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:03.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:05 smithi184 ceph-mon[100622]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:05 smithi184 ceph-mon[102844]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:05 smithi196 ceph-mon[76349]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 82m - - 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 82m - - 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (77m) 17s ago 78m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (77m) 15s ago 77m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:06:07.802 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (71m) 17s ago 76m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (70m) 15s ago 81m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (71m) 17s ago 83m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (70m) 17s ago 83m 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (69m) 15s ago 82m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (69m) 17s ago 82m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (77m) 17s ago 77m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (77m) 15s ago 77m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (68m) 17s ago 81m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (67m) 17s ago 81m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (66m) 17s ago 80m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (66m) 17s ago 80m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (65m) 15s ago 80m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (64m) 15s ago 79m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:06:07.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (63m) 15s ago 79m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (62m) 15s ago 78m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (71m) 15s ago 78m 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (62m) 17s ago 76m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (61m) 15s ago 76m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (61m) 17s ago 76m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:06:07.804 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (61m) 15s ago 76m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:06:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:06:08.113 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:06:08.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:07 smithi184 ceph-mon[100622]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:08.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:07 smithi184 ceph-mon[102844]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:08.383 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:06:08.383 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:06:08.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:06:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:07 smithi196 ceph-mon[76349]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:08.692 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:06:08.692 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:08.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:08.692 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:08.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[100622]: from='client.49008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[100622]: from='client.49014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[100622]: from='client.49020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2551144723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1365723194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[102844]: from='client.49008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[102844]: from='client.49014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[102844]: from='client.49020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2551144723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1365723194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:08 smithi196 ceph-mon[76349]: from='client.49008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:08 smithi196 ceph-mon[76349]: from='client.49014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:08 smithi196 ceph-mon[76349]: from='client.49020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:09.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2551144723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:09.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1365723194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:09 smithi196 ceph-mon[76349]: from='client.49032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:09 smithi196 ceph-mon[76349]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[100622]: from='client.49032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[100622]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[102844]: from='client.49032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[102844]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:10.666 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:10.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:10.666 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:10.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:10.666 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:10.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[102844]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:11 smithi184 ceph-mon[100622]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:11 smithi196 ceph-mon[76349]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:06:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[100622]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-mon[102844]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:13.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:13.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:13 smithi196 ceph-mon[76349]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:14.182 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:13.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:14.182 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:13.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:14.182 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:13.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:14.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:15 smithi184 ceph-mon[102844]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:15 smithi184 ceph-mon[100622]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:15 smithi196 ceph-mon[76349]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:17 smithi196 ceph-mon[76349]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:18.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:17 smithi184 ceph-mon[100622]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:18.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:17 smithi184 ceph-mon[102844]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:19 smithi196 ceph-mon[76349]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:20.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:19 smithi184 ceph-mon[100622]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:20.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:20.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:19 smithi184 ceph-mon[102844]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:20.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:20.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:20.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:20.471 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:20.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:20.471 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:20.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:21 smithi196 ceph-mon[76349]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:21 smithi184 ceph-mon[102844]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:21 smithi184 ceph-mon[100622]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:06:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:23 smithi184 ceph-mon[102844]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:23 smithi184 ceph-mon[100622]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:23.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:23.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:23.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:23 smithi196 ceph-mon[76349]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:25 smithi184 ceph-mon[100622]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:25 smithi184 ceph-mon[102844]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:25 smithi196 ceph-mon[76349]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:27 smithi196 ceph-mon[76349]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:27 smithi184 ceph-mon[100622]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:27 smithi184 ceph-mon[102844]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:29 smithi196 ceph-mon[76349]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:29 smithi184 ceph-mon[100622]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:29 smithi184 ceph-mon[102844]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:30.393Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:30.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:30.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[100622]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:31 smithi184 ceph-mon[102844]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:31 smithi196 ceph-mon[76349]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:06:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[100622]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-mon[102844]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:33 smithi196 ceph-mon[76349]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:33.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:33.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:33.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:35 smithi196 ceph-mon[76349]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:35 smithi184 ceph-mon[100622]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:35 smithi184 ceph-mon[102844]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:37 smithi184 ceph-mon[100622]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:37 smithi184 ceph-mon[102844]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:37 smithi196 ceph-mon[76349]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:39.001 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:06:39.571 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:06:39.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 83m - - 2024-09-20T21:06:39.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 83m - - 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (77m) 8s ago 78m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (78m) 7s ago 78m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (71m) 8s ago 76m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (71m) 7s ago 82m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (72m) 8s ago 84m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (70m) 8s ago 84m 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (70m) 7s ago 82m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (70m) 8s ago 82m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (78m) 8s ago 78m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (78m) 7s ago 78m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (68m) 8s ago 82m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (68m) 8s ago 81m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (67m) 8s ago 81m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:06:39.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (66m) 8s ago 81m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (65m) 7s ago 80m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (64m) 7s ago 80m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (63m) 7s ago 79m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (62m) 7s ago 79m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (71m) 7s ago 79m 69.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (62m) 8s ago 77m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (62m) 7s ago 77m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (62m) 8s ago 76m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:06:39.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (62m) 7s ago 76m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:06:39.889 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:06:39.890 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:06:40.086 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[100622]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:40.087 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:40.087 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3012218130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:40.087 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[102844]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:40.087 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:40.087 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3012218130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:40.175 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:06:40.175 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:06:40.176 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:06:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:39 smithi196 ceph-mon[76349]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3012218130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:40.496 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:06:40.496 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:40.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:40.496 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:40.497 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:40.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:40.394Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:40.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:40.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:40.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:40.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[102844]: from='client.49044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[102844]: from='client.59015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[102844]: from='client.59021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1522824576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[100622]: from='client.49044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[100622]: from='client.59015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[100622]: from='client.59021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1522824576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:40 smithi196 ceph-mon[76349]: from='client.49044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:40 smithi196 ceph-mon[76349]: from='client.59015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:40 smithi196 ceph-mon[76349]: from='client.59021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1522824576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:41 smithi184 ceph-mon[100622]: from='client.59033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:41 smithi184 ceph-mon[100622]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:41 smithi184 ceph-mon[102844]: from='client.59033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:41 smithi184 ceph-mon[102844]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:06:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:41 smithi196 ceph-mon[76349]: from='client.59033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:41 smithi196 ceph-mon[76349]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-mon[100622]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:44.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-mon[102844]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:44.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:44.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:43.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:44.302 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:43.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:44.302 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:43.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:43 smithi196 ceph-mon[76349]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:45 smithi184 ceph-mon[100622]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:45 smithi184 ceph-mon[102844]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:45 smithi196 ceph-mon[76349]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:47 smithi184 ceph-mon[100622]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:47 smithi184 ceph-mon[102844]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:47 smithi196 ceph-mon[76349]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:49 smithi184 ceph-mon[100622]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:49 smithi184 ceph-mon[102844]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:49 smithi196 ceph-mon[76349]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:50.394Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:50.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:50.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:06:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[100622]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[102844]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:51 smithi196 ceph-mon[76349]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[102844]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-mon[100622]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:06:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:53 smithi196 ceph-mon[76349]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:53.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:53.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:06:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:06:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:06:53.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:06:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:55 smithi196 ceph-mon[76349]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:55 smithi184 ceph-mon[102844]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:55 smithi184 ceph-mon[100622]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:57 smithi184 ceph-mon[100622]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:57 smithi184 ceph-mon[102844]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:57 smithi196 ceph-mon[76349]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:59 smithi184 ceph-mon[100622]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:06:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:59 smithi184 ceph-mon[102844]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:06:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:59 smithi196 ceph-mon[76349]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:06:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:00.708 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:00.395Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:00.708 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:00.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:00.708 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:00.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:02.149 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:01 smithi184 ceph-mon[100622]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:02.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:01 smithi184 ceph-mon[102844]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:02.149 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:07:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:01 smithi196 ceph-mon[76349]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:03 smithi184 ceph-mon[100622]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:03 smithi184 ceph-mon[102844]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:03.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:03.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:03.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:03 smithi196 ceph-mon[76349]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:05 smithi184 ceph-mon[100622]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:05 smithi184 ceph-mon[102844]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:05 smithi196 ceph-mon[76349]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:07 smithi196 ceph-mon[76349]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:07 smithi184 ceph-mon[100622]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:07 smithi184 ceph-mon[102844]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:09 smithi184 ceph-mon[100622]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:09 smithi184 ceph-mon[102844]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:09 smithi196 ceph-mon[76349]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:10.793 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:10.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:10.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:10.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:10.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:10.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:10.794 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 83m - - 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 83m - - 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (78m) 0s ago 79m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (78m) 18s ago 78m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (72m) 0s ago 77m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (71m) 18s ago 83m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (72m) 0s ago 84m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (71m) 0s ago 84m 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (70m) 18s ago 83m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (70m) 0s ago 83m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (78m) 0s ago 78m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:07:11.344 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (78m) 18s ago 78m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (69m) 0s ago 82m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (68m) 0s ago 82m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (67m) 0s ago 81m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (67m) 0s ago 81m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (66m) 18s ago 81m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (65m) 18s ago 80m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (64m) 18s ago 80m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (63m) 18s ago 80m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (72m) 18s ago 79m 70.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (63m) 0s ago 77m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (62m) 18s ago 77m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (63m) 0s ago 77m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:07:11.345 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (62m) 18s ago 77m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:07:11.654 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:07:11.924 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:07:11.925 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:07:12.233 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:07:12.233 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:12.234 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:12.234 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:07:12.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: from='client.49080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:12.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:12.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3471944469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: from='client.49080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3471944469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: from='client.49080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3471944469' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='client.59051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='client.49092 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='client.59069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2698881720' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='client.59051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='client.49092 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='client.59069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2698881720' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='client.59051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='client.49092 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='client.59069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2698881720' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-mon[102844]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-mon[100622]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:13.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:13.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:13.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:13 smithi196 ceph-mon[76349]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:16.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:15 smithi196 ceph-mon[76349]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:15 smithi184 ceph-mon[100622]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:15 smithi184 ceph-mon[102844]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:17 smithi196 ceph-mon[76349]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:17 smithi184 ceph-mon[100622]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:17 smithi184 ceph-mon[102844]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:20.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:19 smithi184 ceph-mon[100622]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:20.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:20.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:19 smithi184 ceph-mon[102844]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:20.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:19 smithi196 ceph-mon[76349]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:20.397Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:20.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:20.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:20.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:22.179 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:07:22.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:21 smithi184 ceph-mon[102844]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:22.179 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:21 smithi184 ceph-mon[100622]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:21 smithi196 ceph-mon[76349]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:23 smithi184 ceph-mon[100622]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:23 smithi184 ceph-mon[102844]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:23.834Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:23.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:23.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:23 smithi196 ceph-mon[76349]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:25 smithi184 ceph-mon[100622]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:25 smithi184 ceph-mon[102844]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:25 smithi196 ceph-mon[76349]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:27 smithi184 ceph-mon[100622]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:27 smithi184 ceph-mon[102844]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:27 smithi196 ceph-mon[76349]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:29 smithi196 ceph-mon[76349]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:29 smithi184 ceph-mon[100622]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:29 smithi184 ceph-mon[102844]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:30.398Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:30.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:30.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[100622]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[102844]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:31 smithi196 ceph-mon[76349]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[100622]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-mon[102844]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:33 smithi196 ceph-mon[76349]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:33.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:33.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:33.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:36.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:35 smithi196 ceph-mon[76349]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:35 smithi184 ceph-mon[100622]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:35 smithi184 ceph-mon[102844]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:38.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:37 smithi196 ceph-mon[76349]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:37 smithi184 ceph-mon[100622]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:37 smithi184 ceph-mon[102844]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:39 smithi184 ceph-mon[100622]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:39 smithi184 ceph-mon[102844]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:39 smithi196 ceph-mon[76349]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:40.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:40.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:40.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:42.184 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:41 smithi184 ceph-mon[100622]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:42.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:41 smithi184 ceph-mon[102844]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:42.185 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:07:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:41 smithi196 ceph-mon[76349]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:42.533 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 84m - - 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 84m - - 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (78m) 11s ago 79m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (79m) 10s ago 79m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:07:43.139 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (72m) 11s ago 77m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (72m) 10s ago 83m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (73m) 11s ago 85m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (71m) 11s ago 85m 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (71m) 10s ago 84m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (71m) 11s ago 83m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (79m) 11s ago 79m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (79m) 10s ago 79m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (69m) 11s ago 83m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (69m) 11s ago 82m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (68m) 11s ago 82m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (67m) 11s ago 82m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (66m) 10s ago 81m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (65m) 10s ago 81m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (64m) 10s ago 80m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:07:43.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (64m) 10s ago 80m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:07:43.141 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (72m) 10s ago 80m 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:07:43.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (63m) 11s ago 78m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:07:43.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (63m) 10s ago 78m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:07:43.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (63m) 11s ago 77m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:07:43.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (63m) 10s ago 77m 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:07:43.451 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:07:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:07:43.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:07:43.453 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:07:43.727 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:07:43.728 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:07:44.041 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:07:44.064 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:44.064 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:44.065 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:44.065 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:44.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[100622]: from='client.59081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[100622]: from='client.59087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[100622]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:44.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1503025445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:44.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:44.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[102844]: from='client.59081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[102844]: from='client.59087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[102844]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:44.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1503025445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:44.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:44.066 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:43.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:44.066 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:43.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:44.066 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:43.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:43 smithi196 ceph-mon[76349]: from='client.59081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:43 smithi196 ceph-mon[76349]: from='client.59087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:43 smithi196 ceph-mon[76349]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1503025445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[100622]: from='client.59093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[100622]: from='client.49140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2698970510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[102844]: from='client.59093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[102844]: from='client.49140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2698970510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:44 smithi196 ceph-mon[76349]: from='client.59093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:44 smithi196 ceph-mon[76349]: from='client.49140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2698970510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:45 smithi184 ceph-mon[100622]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:45 smithi184 ceph-mon[102844]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:45 smithi196 ceph-mon[76349]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:47 smithi184 ceph-mon[100622]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:47 smithi184 ceph-mon[102844]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:47 smithi196 ceph-mon[76349]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:49 smithi184 ceph-mon[100622]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:49 smithi184 ceph-mon[102844]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:49 smithi196 ceph-mon[76349]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:50.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded" 2024-09-20T21:07:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:50.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:50.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:07:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[100622]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[102844]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:51 smithi196 ceph-mon[76349]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:53 smithi196 ceph-mon[76349]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[102844]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:07:54.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-mon[100622]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:53.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:53.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:07:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:07:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:07:53.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:07:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:55 smithi184 ceph-mon[100622]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:55 smithi184 ceph-mon[102844]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:56.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:55 smithi196 ceph-mon[76349]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:57 smithi184 ceph-mon[100622]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:57 smithi184 ceph-mon[102844]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:57 smithi196 ceph-mon[76349]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:59 smithi184 ceph-mon[100622]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:07:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:59 smithi184 ceph-mon[102844]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:07:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:59 smithi196 ceph-mon[76349]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:07:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:00.400Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:00.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:00.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:02.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:01 smithi184 ceph-mon[100622]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:02.254 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:01 smithi184 ceph-mon[102844]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:02.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:08:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:01 smithi196 ceph-mon[76349]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:04.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:03 smithi184 ceph-mon[100622]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:04.141 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:03.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:04.141 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:03.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:04.141 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:03.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:04.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:03 smithi184 ceph-mon[102844]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:03 smithi196 ceph-mon[76349]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:08:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:05 smithi184 ceph-mon[100622]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:05 smithi184 ceph-mon[102844]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:05 smithi196 ceph-mon[76349]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:07 smithi184 ceph-mon[100622]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:07 smithi184 ceph-mon[102844]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:07 smithi196 ceph-mon[76349]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:09 smithi184 ceph-mon[100622]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:09 smithi184 ceph-mon[102844]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:09 smithi196 ceph-mon[76349]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:10.401Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:10.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:10.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[100622]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[102844]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:11 smithi196 ceph-mon[76349]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[100622]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[102844]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:13.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:13.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:14.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:13.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-mon[76349]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:14.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:08:13.812320168Z level=info msg="Completed cleanup jobs" duration=12.833729ms 2024-09-20T21:08:14.338 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 84m - - 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 84m - - 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (79m) 3s ago 80m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (80m) 1s ago 80m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (73m) 3s ago 78m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:08:14.866 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (73m) 1s ago 84m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (73m) 3s ago 85m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (72m) 3s ago 85m 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (71m) 1s ago 84m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (71m) 3s ago 84m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (79m) 3s ago 79m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (79m) 1s ago 79m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (70m) 3s ago 83m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (69m) 3s ago 83m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (68m) 3s ago 83m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (68m) 3s ago 82m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (67m) 1s ago 82m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (66m) 1s ago 81m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:08:14.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (65m) 1s ago 81m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (64m) 1s ago 81m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (73m) 1s ago 80m 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (64m) 3s ago 78m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (63m) 1s ago 78m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (64m) 3s ago 78m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:08:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (63m) 1s ago 78m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:15.183 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:08:15.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:08:15.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:15.184 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:08:15.185 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:08:15.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:08:15.185 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:08:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:08:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:08:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:08:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:08:15.464 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:08:15.773 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:08:15.774 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:15.774 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:15.774 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:15.774 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: from='client.49152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: from='client.49158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: from='client.59129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2926957539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1910576800' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: from='client.49152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: from='client.49158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: from='client.59129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2926957539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1910576800' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: from='client.49152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: from='client.49158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: from='client.59129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2926957539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1910576800' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:17.098 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:16 smithi196 ceph-mon[76349]: from='client.49176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:16 smithi184 ceph-mon[100622]: from='client.49176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:16 smithi184 ceph-mon[102844]: from='client.49176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:18.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:17 smithi196 ceph-mon[76349]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:17 smithi184 ceph-mon[100622]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:17 smithi184 ceph-mon[102844]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:19 smithi196 ceph-mon[76349]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:19 smithi184 ceph-mon[100622]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:19 smithi184 ceph-mon[102844]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:20.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:20.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:20.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:22.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:21 smithi184 ceph-mon[100622]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:22.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:21 smithi184 ceph-mon[102844]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:22.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:08:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:21 smithi196 ceph-mon[76349]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:23 smithi184 ceph-mon[100622]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:23 smithi184 ceph-mon[102844]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:23.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:23.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:23.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:23 smithi196 ceph-mon[76349]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:25 smithi184 ceph-mon[100622]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:25 smithi184 ceph-mon[102844]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:25 smithi196 ceph-mon[76349]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:27 smithi184 ceph-mon[100622]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:27 smithi184 ceph-mon[102844]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:27 smithi196 ceph-mon[76349]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:29 smithi184 ceph-mon[100622]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:29 smithi184 ceph-mon[102844]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:29 smithi196 ceph-mon[76349]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:30.403Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:30.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:30.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[100622]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[102844]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:31 smithi196 ceph-mon[76349]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[100622]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[102844]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:33.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:33.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:33.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:33 smithi196 ceph-mon[76349]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:35 smithi184 ceph-mon[102844]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:35 smithi184 ceph-mon[100622]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:35 smithi196 ceph-mon[76349]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:38.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:37 smithi196 ceph-mon[76349]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:37 smithi184 ceph-mon[100622]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:37 smithi184 ceph-mon[102844]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:39 smithi184 ceph-mon[100622]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:39 smithi184 ceph-mon[102844]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:39 smithi196 ceph-mon[76349]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:40.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:40.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:40.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:41 smithi184 ceph-mon[100622]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:41 smithi184 ceph-mon[102844]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:08:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:41 smithi196 ceph-mon[76349]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:43.997 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:43.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:43.997 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:43.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:43.997 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:43.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:44.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-mon[100622]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:44.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:44.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-mon[102844]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:44.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:43 smithi196 ceph-mon[76349]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:46.072 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:08:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:45 smithi184 ceph-mon[100622]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:45 smithi184 ceph-mon[102844]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:45 smithi196 ceph-mon[76349]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:46.602 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:08:46.602 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 85m - - 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 85m - - 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (79m) 15s ago 80m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (80m) 13s ago 80m 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (73m) 15s ago 78m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (73m) 13s ago 84m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (74m) 15s ago 86m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (72m) 15s ago 86m 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (72m) 13s ago 85m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (72m) 15s ago 84m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (80m) 15s ago 80m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (80m) 13s ago 80m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (71m) 15s ago 84m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (70m) 15s ago 83m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:08:46.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (69m) 15s ago 83m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (69m) 15s ago 83m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (67m) 13s ago 82m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (66m) 13s ago 82m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (65m) 13s ago 82m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (65m) 13s ago 81m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (73m) 13s ago 81m 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (64m) 15s ago 79m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (64m) 13s ago 79m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (64m) 15s ago 78m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:08:46.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (64m) 13s ago 78m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:08:46.912 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:08:46.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:08:46.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:08:46.914 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:08:47.186 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:08:47.187 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:08:47.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/642489884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:47.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/642489884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/642489884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:47.496 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:08:47.496 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:47.496 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[100622]: from='client.59153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[100622]: from='client.49194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[100622]: from='client.59165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[100622]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2630916484' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[102844]: from='client.59153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[102844]: from='client.49194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[102844]: from='client.59165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[102844]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2630916484' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:48 smithi196 ceph-mon[76349]: from='client.59153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:48 smithi196 ceph-mon[76349]: from='client.49194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:48 smithi196 ceph-mon[76349]: from='client.59165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:48 smithi196 ceph-mon[76349]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2630916484' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[100622]: from='client.49212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[100622]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[102844]: from='client.49212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[102844]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:49 smithi196 ceph-mon[76349]: from='client.49212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:49 smithi196 ceph-mon[76349]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:50.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:50.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:50.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[100622]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:08:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[102844]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:51 smithi196 ceph-mon[76349]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[100622]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[102844]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:53.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:53.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:08:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:08:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:08:53.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:08:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:53 smithi196 ceph-mon[76349]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:08:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:55 smithi184 ceph-mon[100622]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:55 smithi184 ceph-mon[102844]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:55 smithi196 ceph-mon[76349]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:58.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:57 smithi196 ceph-mon[76349]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:57 smithi184 ceph-mon[100622]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:57 smithi184 ceph-mon[102844]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:59.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:59 smithi196 ceph-mon[76349]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:08:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:59 smithi184 ceph-mon[100622]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:08:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:59 smithi184 ceph-mon[102844]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:08:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:00.405Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:00.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:00.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:01 smithi184 ceph-mon[100622]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:01 smithi184 ceph-mon[102844]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:09:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:01 smithi196 ceph-mon[76349]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:03 smithi184 ceph-mon[100622]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:03 smithi184 ceph-mon[102844]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:03.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:03.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:04.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:03.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:03 smithi196 ceph-mon[76349]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:06.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:05 smithi184 ceph-mon[102844]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:06.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:05 smithi184 ceph-mon[100622]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:05 smithi196 ceph-mon[76349]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:07 smithi184 ceph-mon[100622]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:07 smithi184 ceph-mon[102844]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:07 smithi196 ceph-mon[76349]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:09 smithi184 ceph-mon[100622]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:09 smithi184 ceph-mon[102844]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:09 smithi196 ceph-mon[76349]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:10.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:10.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:10.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[100622]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[102844]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:11 smithi196 ceph-mon[76349]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[100622]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[102844]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:13.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:13.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:13.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:13 smithi196 ceph-mon[76349]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:15 smithi184 ceph-mon[100622]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:15 smithi184 ceph-mon[102844]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:15 smithi196 ceph-mon[76349]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:17.793 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:09:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:17 smithi184 ceph-mon[100622]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:17 smithi184 ceph-mon[102844]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 86m - - 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 85m - - 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (80m) 6s ago 81m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (81m) 4s ago 81m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (74m) 6s ago 79m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (74m) 4s ago 85m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (74m) 6s ago 86m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:09:18.330 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (73m) 6s ago 86m 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (72m) 4s ago 85m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (73m) 6s ago 85m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (80m) 6s ago 80m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (80m) 4s ago 80m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (71m) 6s ago 84m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (70m) 6s ago 84m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (70m) 6s ago 84m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (69m) 6s ago 83m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (68m) 4s ago 83m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (67m) 4s ago 82m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (66m) 4s ago 82m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (65m) 4s ago 82m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (74m) 4s ago 81m 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:09:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (65m) 6s ago 79m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:09:18.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (65m) 4s ago 79m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:09:18.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (65m) 6s ago 79m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:09:18.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (64m) 4s ago 79m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:09:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:17 smithi196 ceph-mon[76349]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:09:18.639 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:09:18.640 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:09:18.909 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:09:18.910 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:09:18.910 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:09:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:18 smithi196 ceph-mon[76349]: from='client.59189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/471130947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:19.224 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:09:19.224 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:19.224 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:19.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:18 smithi184 ceph-mon[100622]: from='client.59189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:19.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/471130947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:19.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:18 smithi184 ceph-mon[102844]: from='client.59189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:19.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/471130947' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: from='client.49230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: from='client.49236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: from='client.59213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/931906674' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: from='client.49230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: from='client.49236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: from='client.59213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/931906674' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: from='client.49230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: from='client.49236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: from='client.59213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:20.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/931906674' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:20.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:20.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:20.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:21 smithi184 ceph-mon[100622]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:21 smithi184 ceph-mon[102844]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:09:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:21 smithi196 ceph-mon[76349]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:23 smithi184 ceph-mon[100622]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:23 smithi184 ceph-mon[102844]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:23.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:23.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:23.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:23 smithi196 ceph-mon[76349]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:25 smithi184 ceph-mon[100622]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:25 smithi184 ceph-mon[102844]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:25 smithi196 ceph-mon[76349]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:27 smithi184 ceph-mon[100622]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:27 smithi184 ceph-mon[102844]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:28 smithi196 ceph-mon[76349]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-mon[102844]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-mon[100622]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:30.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:30 smithi196 ceph-mon[76349]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:30.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:30.407Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:30.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:30.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[100622]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[102844]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:32 smithi196 ceph-mon[76349]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[100622]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[102844]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:33.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:33.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:33.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:34 smithi196 ceph-mon[76349]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:35 smithi196 ceph-mon[76349]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:35 smithi184 ceph-mon[100622]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:35 smithi184 ceph-mon[102844]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:37 smithi184 ceph-mon[100622]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:37 smithi184 ceph-mon[102844]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:37 smithi196 ceph-mon[76349]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:39 smithi184 ceph-mon[100622]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:39 smithi184 ceph-mon[102844]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:39 smithi196 ceph-mon[76349]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:40.408Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:40.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:40.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:41 smithi196 ceph-mon[76349]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:41 smithi184 ceph-mon[100622]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:41 smithi184 ceph-mon[102844]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:09:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-mon[100622]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-mon[102844]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:43.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:43.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:43.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:43 smithi196 ceph-mon[76349]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:46.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:45 smithi184 ceph-mon[100622]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:46.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:45 smithi184 ceph-mon[102844]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:45 smithi196 ceph-mon[76349]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:47 smithi184 ceph-mon[100622]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:47 smithi184 ceph-mon[102844]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:47 smithi196 ceph-mon[76349]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:49.513 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:09:50.038 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 86m - - 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 86m - - 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (80m) 18s ago 81m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (81m) 16s ago 81m 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (74m) 18s ago 79m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (74m) 16s ago 85m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (75m) 18s ago 87m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (73m) 18s ago 87m 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (73m) 16s ago 86m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (73m) 18s ago 85m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:09:50.039 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (81m) 18s ago 81m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (81m) 16s ago 81m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (72m) 18s ago 85m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (71m) 18s ago 85m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (70m) 18s ago 84m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (70m) 18s ago 84m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (68m) 16s ago 83m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (67m) 16s ago 83m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (66m) 16s ago 83m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (66m) 16s ago 82m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (74m) 16s ago 82m 73.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (65m) 18s ago 80m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (65m) 16s ago 80m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (65m) 18s ago 79m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:09:50.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (65m) 16s ago 79m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:09:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:49 smithi184 ceph-mon[100622]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:49 smithi184 ceph-mon[102844]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:50.346 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:09:50.346 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:09:50.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:09:50.347 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:09:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:49 smithi196 ceph-mon[76349]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:09:50.612 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:09:50.613 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:09:50.613 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:09:50.613 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:09:50.613 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:50.409Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:50.613 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:50.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:50.613 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:50.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:50.921 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:09:50.921 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:50.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:50.921 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:50.922 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[100622]: from='client.49260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[100622]: from='client.59231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/13293759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/805639946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[102844]: from='client.49260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[102844]: from='client.59231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/13293759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/805639946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:51.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:51.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:50 smithi196 ceph-mon[76349]: from='client.49260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:50 smithi196 ceph-mon[76349]: from='client.59231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:51.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/13293759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:51.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/805639946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:09:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[100622]: from='client.49272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[100622]: from='client.59249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[100622]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[102844]: from='client.49272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[102844]: from='client.59249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:51 smithi184 ceph-mon[102844]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:51 smithi196 ceph-mon[76349]: from='client.49272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:51 smithi196 ceph-mon[76349]: from='client.59249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:51 smithi196 ceph-mon[76349]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:53 smithi196 ceph-mon[76349]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[100622]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-mon[102844]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:53.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:53.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:09:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:09:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:09:53.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:09:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:09:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:55 smithi196 ceph-mon[76349]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:55 smithi184 ceph-mon[100622]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:55 smithi184 ceph-mon[102844]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:57 smithi184 ceph-mon[100622]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:57 smithi184 ceph-mon[102844]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:57 smithi196 ceph-mon[76349]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:00.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:09:59 smithi196 ceph-mon[76349]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:09:59 smithi184 ceph-mon[100622]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:10:00.000+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:09:59 smithi184 ceph-mon[102844]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:00.410Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:00.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:00.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:01 smithi184 ceph-mon[100622]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:01 smithi184 ceph-mon[102844]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:10:02.351 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:01 smithi196 ceph-mon[76349]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:03.975 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:03.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:03.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:03.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:03.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:03.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:03 smithi184 ceph-mon[100622]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:03 smithi184 ceph-mon[102844]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:03 smithi196 ceph-mon[76349]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:05 smithi184 ceph-mon[100622]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:05 smithi184 ceph-mon[102844]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:05 smithi196 ceph-mon[76349]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:07 smithi184 ceph-mon[100622]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:07 smithi184 ceph-mon[102844]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:07 smithi196 ceph-mon[76349]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:09 smithi184 ceph-mon[100622]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:09 smithi184 ceph-mon[102844]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:09 smithi196 ceph-mon[76349]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:10.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:10.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:10.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:11 smithi184 ceph-mon[100622]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:11 smithi184 ceph-mon[102844]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:10:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:11 smithi196 ceph-mon[76349]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[100622]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-mon[102844]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:13 smithi196 ceph-mon[76349]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:13.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:13.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:13.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:15 smithi184 ceph-mon[100622]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:15 smithi184 ceph-mon[102844]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:15 smithi196 ceph-mon[76349]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:17 smithi184 ceph-mon[100622]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:17 smithi184 ceph-mon[102844]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:17 smithi196 ceph-mon[76349]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:19 smithi184 ceph-mon[100622]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:19 smithi184 ceph-mon[102844]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:19 smithi196 ceph-mon[76349]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:20.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:20.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:20.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:21.219 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:10:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 87m - - 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 86m - - 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (81m) 9s ago 82m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (82m) 8s ago 82m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (75m) 9s ago 80m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (75m) 8s ago 86m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (76m) 9s ago 87m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (74m) 9s ago 88m 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (73m) 8s ago 86m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:10:21.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (74m) 9s ago 86m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (81m) 9s ago 81m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (81m) 8s ago 81m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (72m) 9s ago 85m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (71m) 9s ago 85m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (71m) 9s ago 85m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (70m) 9s ago 84m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (69m) 8s ago 84m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (68m) 8s ago 83m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (67m) 8s ago 83m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (66m) 8s ago 83m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (75m) 8s ago 82m 73.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (66m) 9s ago 80m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (66m) 8s ago 80m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (66m) 9s ago 80m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:10:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (65m) 8s ago 80m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:10:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:21 smithi184 ceph-mon[100622]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:21 smithi184 ceph-mon[102844]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:22.057 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:10:22.057 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:10:22.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:10:22.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:10:22.059 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:10:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:21 smithi196 ceph-mon[76349]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:10:22.327 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:10:22.636 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:22.636 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:22.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:22.636 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:22.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[100622]: from='client.49296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[100622]: from='client.49302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[100622]: from='client.49308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2857416916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3119717606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[102844]: from='client.49296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[102844]: from='client.49302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[102844]: from='client.49308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2857416916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:23.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3119717606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:23.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:22 smithi196 ceph-mon[76349]: from='client.49296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:22 smithi196 ceph-mon[76349]: from='client.49302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:22 smithi196 ceph-mon[76349]: from='client.49308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:23.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2857416916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:23.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3119717606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-mon[100622]: from='client.49320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-mon[100622]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-mon[102844]: from='client.49320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-mon[102844]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:23.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:23.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:23.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:23 smithi196 ceph-mon[76349]: from='client.49320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:23 smithi196 ceph-mon[76349]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:25 smithi184 ceph-mon[100622]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:25 smithi184 ceph-mon[102844]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:25 smithi196 ceph-mon[76349]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:27.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:27 smithi184 ceph-mon[100622]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:27.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:27 smithi184 ceph-mon[102844]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:27 smithi196 ceph-mon[76349]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:29 smithi196 ceph-mon[76349]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:29 smithi184 ceph-mon[100622]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:29 smithi184 ceph-mon[102844]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:30.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:30.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:30.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:31 smithi196 ceph-mon[76349]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:31 smithi184 ceph-mon[100622]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:10:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:31 smithi184 ceph-mon[102844]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:33 smithi196 ceph-mon[76349]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[100622]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-mon[102844]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:33.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:33.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:33.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:35 smithi196 ceph-mon[76349]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:35 smithi184 ceph-mon[100622]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:35 smithi184 ceph-mon[102844]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:37 smithi184 ceph-mon[100622]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:37 smithi184 ceph-mon[102844]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:37 smithi196 ceph-mon[76349]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:39 smithi184 ceph-mon[100622]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:39 smithi184 ceph-mon[102844]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:39 smithi196 ceph-mon[76349]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:40.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:40.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:40.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:41.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:41 smithi196 ceph-mon[76349]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:41 smithi184 ceph-mon[100622]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:41 smithi184 ceph-mon[102844]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:10:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-mon[100622]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-mon[102844]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:43.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:43.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:43.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:43 smithi196 ceph-mon[76349]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:45 smithi184 ceph-mon[100622]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:45 smithi184 ceph-mon[102844]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:45 smithi196 ceph-mon[76349]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:48.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:47 smithi184 ceph-mon[100622]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:48.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:47 smithi184 ceph-mon[102844]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:47 smithi196 ceph-mon[76349]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:49 smithi184 ceph-mon[100622]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:49 smithi184 ceph-mon[102844]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:49 smithi196 ceph-mon[76349]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:50.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:50.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:50.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:52 smithi184 ceph-mon[100622]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:52 smithi184 ceph-mon[102844]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:10:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:10:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:52 smithi196 ceph-mon[76349]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:52.927 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:10:53.457 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 87m - - 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 87m - - 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (81m) 1s ago 82m 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (82m) 19s ago 82m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (75m) 1s ago 80m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (75m) 19s ago 86m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (76m) 1s ago 88m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (74m) 1s ago 88m 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (74m) 19s ago 87m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:10:53.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (74m) 1s ago 87m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (82m) 1s ago 82m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (82m) 19s ago 82m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (73m) 1s ago 86m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (72m) 1s ago 86m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (71m) 1s ago 85m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (71m) 1s ago 85m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (69m) 19s ago 84m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (68m) 19s ago 84m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (67m) 19s ago 84m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (67m) 19s ago 83m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (76m) 19s ago 83m 73.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (66m) 1s ago 81m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (66m) 19s ago 81m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:10:53.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (66m) 1s ago 81m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:10:53.460 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (66m) 19s ago 80m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:10:53.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:10:53.771 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:10:53.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[102844]: from='client.49332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[102844]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[100622]: from='client.49332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:53.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-mon[100622]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:53 smithi196 ceph-mon[76349]: from='client.49332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:53 smithi196 ceph-mon[76349]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:10:54.042 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:10:54.043 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:10:54.043 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:10:54.043 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:10:54.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:10:54.043 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:53.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:54.043 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:53.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:10:54.043 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:10:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:10:53.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:10:54.352 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:54.352 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:54.352 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:54.352 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:54.352 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='client.49338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='client.59309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3935702826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/595923452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='client.49338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='client.59309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3935702826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/595923452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='client.49338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='client.59309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3935702826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:54.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:10:54.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/595923452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:55 smithi184 ceph-mon[102844]: from='client.49356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:55 smithi184 ceph-mon[102844]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:55 smithi184 ceph-mon[100622]: from='client.49356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:55 smithi184 ceph-mon[100622]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:55 smithi196 ceph-mon[76349]: from='client.49356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:55 smithi196 ceph-mon[76349]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:57 smithi184 ceph-mon[100622]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:57 smithi184 ceph-mon[102844]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:57 smithi196 ceph-mon[76349]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:10:59 smithi184 ceph-mon[100622]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:10:59 smithi184 ceph-mon[102844]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:10:59 smithi196 ceph-mon[76349]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:00.414Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:00.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:00.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:01 smithi184 ceph-mon[100622]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:01 smithi184 ceph-mon[102844]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T21:11:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:01 smithi196 ceph-mon[76349]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:04.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:03 smithi196 ceph-mon[76349]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:03 smithi184 ceph-mon[100622]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:03 smithi184 ceph-mon[102844]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:03.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:03.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:03.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:05 smithi184 ceph-mon[100622]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:06 smithi184 ceph-mon[102844]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:05 smithi196 ceph-mon[76349]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:08 smithi184 ceph-mon[100622]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:08 smithi184 ceph-mon[102844]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:08 smithi196 ceph-mon[76349]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:10 smithi184 ceph-mon[100622]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:10 smithi184 ceph-mon[102844]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:10 smithi196 ceph-mon[76349]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:10.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:10.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:10.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:12 smithi184 ceph-mon[100622]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T21:11:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:12 smithi184 ceph-mon[102844]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:12 smithi196 ceph-mon[76349]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:13.853 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[102844]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.854 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-mon[100622]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:13 smithi196 ceph-mon[76349]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:13.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:13.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:13.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:15 smithi196 ceph-mon[76349]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:15 smithi184 ceph-mon[100622]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:15 smithi184 ceph-mon[102844]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:17 smithi184 ceph-mon[100622]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:17 smithi184 ceph-mon[102844]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:17 smithi196 ceph-mon[76349]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:19 smithi184 ceph-mon[100622]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:19 smithi184 ceph-mon[102844]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:19 smithi196 ceph-mon[76349]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:20.416Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:20.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:20.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:22.248 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:21 smithi196 ceph-mon[76349]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:21 smithi184 ceph-mon[100622]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:21 smithi184 ceph-mon[102844]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:11:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:23 smithi184 ceph-mon[100622]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:23 smithi184 ceph-mon[102844]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:23.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:23.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:23.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:23 smithi196 ceph-mon[76349]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:11:25.178 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:11:25.178 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 88m - - 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 87m - - 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (82m) 12s ago 83m 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (83m) 11s ago 83m 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (76m) 12s ago 81m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (76m) 11s ago 87m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (77m) 12s ago 88m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (75m) 12s ago 89m 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (74m) 11s ago 87m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (75m) 12s ago 87m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (83m) 12s ago 82m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:11:25.179 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (82m) 11s ago 82m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (73m) 12s ago 86m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (72m) 12s ago 86m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (72m) 12s ago 86m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (71m) 12s ago 85m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (70m) 11s ago 85m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (69m) 11s ago 85m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (68m) 11s ago 84m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (67m) 11s ago 84m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (76m) 11s ago 83m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (67m) 12s ago 81m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:11:25.180 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (67m) 11s ago 82m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:11:25.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (67m) 12s ago 81m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:11:25.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (67m) 11s ago 81m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:11:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:11:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:11:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:11:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:11:25.493 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:11:25.780 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:11:25.781 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:11:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[100622]: from='client.59333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[100622]: from='client.59339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[100622]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3627242190' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[102844]: from='client.59333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[102844]: from='client.59339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[102844]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3627242190' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:26.092 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:11:26.092 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:26.092 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:26 smithi196 ceph-mon[76349]: from='client.59333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:26 smithi196 ceph-mon[76349]: from='client.59339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:26 smithi196 ceph-mon[76349]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3627242190' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[100622]: from='client.49380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[100622]: from='client.49392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1279111486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[102844]: from='client.49380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[102844]: from='client.49392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1279111486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:27 smithi196 ceph-mon[76349]: from='client.49380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:27 smithi196 ceph-mon[76349]: from='client.49392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1279111486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:28.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:28 smithi184 ceph-mon[100622]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:28.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:28 smithi184 ceph-mon[102844]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:28 smithi196 ceph-mon[76349]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:29.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:29.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:30 smithi184 ceph-mon[100622]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:30 smithi184 ceph-mon[102844]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:30.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:30 smithi196 ceph-mon[76349]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:30.416Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:30.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:30.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:32 smithi184 ceph-mon[100622]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:32 smithi184 ceph-mon[102844]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:11:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:32 smithi196 ceph-mon[76349]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[102844]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-mon[100622]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:33.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:33.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:34.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:33.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:33 smithi196 ceph-mon[76349]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[100622]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:35 smithi184 ceph-mon[102844]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:35 smithi196 ceph-mon[76349]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:37 smithi184 ceph-mon[100622]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:37 smithi184 ceph-mon[102844]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:37 smithi196 ceph-mon[76349]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:39 smithi184 ceph-mon[100622]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:39 smithi184 ceph-mon[102844]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:39 smithi196 ceph-mon[76349]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:40.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:40.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:40.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:42 smithi184 ceph-mon[100622]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:11:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:42 smithi184 ceph-mon[102844]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:42 smithi196 ceph-mon[76349]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:44.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:44 smithi196 ceph-mon[76349]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:44.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:44 smithi184 ceph-mon[100622]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:44 smithi184 ceph-mon[102844]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:43.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:43.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:43.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:46 smithi196 ceph-mon[76349]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:46 smithi184 ceph-mon[100622]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:46 smithi184 ceph-mon[102844]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:48 smithi184 ceph-mon[100622]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:48 smithi184 ceph-mon[102844]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:48 smithi196 ceph-mon[76349]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:50 smithi184 ceph-mon[100622]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:50 smithi184 ceph-mon[102844]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:50 smithi196 ceph-mon[76349]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:50.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:50.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:50.634 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:50.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:50.634 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:50.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:52 smithi184 ceph-mon[100622]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:52 smithi184 ceph-mon[102844]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:11:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:11:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:52 smithi196 ceph-mon[76349]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:53 smithi196 ceph-mon[76349]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[100622]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-mon[102844]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:53.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:53.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:11:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:11:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:11:53.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:55 smithi196 ceph-mon[76349]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[100622]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:11:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:55 smithi184 ceph-mon[102844]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:56.387 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:11:56.917 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:11:56.917 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 88m - - 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 88m - - 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (82m) 4s ago 83m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (83m) 2s ago 83m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (76m) 4s ago 81m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (76m) 2s ago 87m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:11:56.918 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (77m) 4s ago 89m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (75m) 4s ago 89m 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (75m) 2s ago 88m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (75m) 4s ago 88m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (83m) 4s ago 83m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (83m) 2s ago 83m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:11:56.919 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (74m) 4s ago 87m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (73m) 4s ago 87m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (72m) 4s ago 86m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (72m) 4s ago 86m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (70m) 2s ago 85m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (70m) 2s ago 85m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:11:56.920 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (68m) 2s ago 85m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (68m) 2s ago 84m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (77m) 2s ago 84m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (67m) 4s ago 82m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (67m) 2s ago 82m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (67m) 4s ago 82m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:11:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (67m) 2s ago 82m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:11:57.227 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:11:57.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:11:57.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:11:57.229 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:11:57.229 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:11:57.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:11:57.229 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:11:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:11:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:11:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:11:57.498 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:11:57.808 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:11:57.809 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:57.809 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:57.809 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:57.809 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: from='client.49404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: from='client.49410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: from='client.59381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/934244728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3580821607' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: from='client.49404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: from='client.49410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: from='client.59381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/934244728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3580821607' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: from='client.49404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: from='client.49410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: from='client.59381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/934244728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:58.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3580821607' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:59 smithi196 ceph-mon[76349]: from='client.49428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:59 smithi184 ceph-mon[100622]: from='client.49428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:59 smithi184 ceph-mon[102844]: from='client.49428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:00 smithi184 ceph-mon[100622]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:00 smithi184 ceph-mon[102844]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:00 smithi196 ceph-mon[76349]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:00.418Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:00.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:00.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:02 smithi184 ceph-mon[100622]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:02 smithi184 ceph-mon[102844]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:12:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:02 smithi196 ceph-mon[76349]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:04 smithi184 ceph-mon[100622]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:04 smithi184 ceph-mon[102844]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:03.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:03.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:03.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:04 smithi196 ceph-mon[76349]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:06 smithi196 ceph-mon[76349]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:06 smithi184 ceph-mon[100622]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:06 smithi184 ceph-mon[102844]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:08 smithi184 ceph-mon[100622]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:08 smithi184 ceph-mon[102844]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:08 smithi196 ceph-mon[76349]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:10 smithi184 ceph-mon[100622]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:10 smithi184 ceph-mon[102844]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:10 smithi196 ceph-mon[76349]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:10.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:10.419Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:10.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:10.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:10.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:10.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:12.031 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:12:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:12 smithi184 ceph-mon[100622]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:12 smithi184 ceph-mon[102844]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:12 smithi196 ceph-mon[76349]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[102844]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:14.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[100622]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:14.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:13.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:14.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:13.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:14.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:13.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:13 smithi196 ceph-mon[76349]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[100622]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:15 smithi184 ceph-mon[102844]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:15 smithi196 ceph-mon[76349]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:18 smithi184 ceph-mon[100622]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:18 smithi184 ceph-mon[102844]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:18 smithi196 ceph-mon[76349]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:20 smithi184 ceph-mon[100622]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:20 smithi184 ceph-mon[102844]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:20 smithi196 ceph-mon[76349]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:20.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:20.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:20.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:22 smithi184 ceph-mon[100622]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:12:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:22 smithi184 ceph-mon[102844]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:22 smithi196 ceph-mon[76349]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:24 smithi184 ceph-mon[100622]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:24 smithi184 ceph-mon[102844]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:23.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:23.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:23.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:24.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:24 smithi196 ceph-mon[76349]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:26 smithi184 ceph-mon[100622]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:26 smithi184 ceph-mon[102844]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:26.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:26 smithi196 ceph-mon[76349]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:28.107 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:12:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:28 smithi184 ceph-mon[100622]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:28 smithi184 ceph-mon[102844]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:28 smithi196 ceph-mon[76349]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:12:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 89m - - 2024-09-20T21:12:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 88m - - 2024-09-20T21:12:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (83m) 15s ago 84m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (84m) 14s ago 84m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (77m) 15s ago 82m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (77m) 14s ago 88m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (78m) 15s ago 90m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (76m) 15s ago 90m 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (75m) 14s ago 88m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (76m) 15s ago 88m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (84m) 15s ago 84m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (83m) 14s ago 83m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (74m) 15s ago 88m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (73m) 15s ago 87m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (73m) 15s ago 87m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (72m) 15s ago 86m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (71m) 14s ago 86m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:12:28.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (70m) 14s ago 86m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (69m) 14s ago 85m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (68m) 14s ago 85m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (77m) 14s ago 84m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (68m) 15s ago 82m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (68m) 14s ago 83m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (68m) 15s ago 82m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:12:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (68m) 14s ago 82m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:12:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:12:28.951 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:12:29.220 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:12:29.221 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:12:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:12:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:12:29.222 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:12:29.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:29.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/340255041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:29.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:29.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/340255041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/340255041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:29.530 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:12:29.531 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:29.531 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:29.531 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:29.531 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:30.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[100622]: from='client.49440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[100622]: from='client.59411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[100622]: from='client.59417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[100622]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:30.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3519256707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:30.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[102844]: from='client.49440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[102844]: from='client.59411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[102844]: from='client.59417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[102844]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:30.138 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3519256707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:30 smithi196 ceph-mon[76349]: from='client.49440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:30 smithi196 ceph-mon[76349]: from='client.59411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:30 smithi196 ceph-mon[76349]: from='client.59417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:30 smithi196 ceph-mon[76349]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3519256707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:30.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:30.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:30.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:31.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:31.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:31 smithi184 ceph-mon[100622]: from='client.49464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:31.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:31.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:31 smithi184 ceph-mon[102844]: from='client.49464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:31 smithi196 ceph-mon[76349]: from='client.49464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:32 smithi184 ceph-mon[102844]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:32 smithi184 ceph-mon[100622]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:12:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:32 smithi196 ceph-mon[76349]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[102844]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[100622]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:34.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:33.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:33.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:34.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:33.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:34 smithi196 ceph-mon[76349]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:35 smithi196 ceph-mon[76349]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[100622]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:35 smithi184 ceph-mon[102844]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:38 smithi184 ceph-mon[100622]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:38 smithi184 ceph-mon[102844]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:38 smithi196 ceph-mon[76349]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:40 smithi184 ceph-mon[100622]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:40 smithi184 ceph-mon[102844]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:40 smithi196 ceph-mon[76349]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:40.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:40.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:40.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:42 smithi184 ceph-mon[100622]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:42 smithi184 ceph-mon[102844]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:12:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:42 smithi196 ceph-mon[76349]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:44 smithi184 ceph-mon[100622]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:44 smithi184 ceph-mon[102844]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:43.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:43.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:43.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:44 smithi196 ceph-mon[76349]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:46.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:46 smithi196 ceph-mon[76349]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:46 smithi184 ceph-mon[100622]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:46 smithi184 ceph-mon[102844]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:48 smithi184 ceph-mon[100622]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:48 smithi184 ceph-mon[102844]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:48 smithi196 ceph-mon[76349]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:50 smithi184 ceph-mon[100622]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:50 smithi184 ceph-mon[102844]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:50 smithi196 ceph-mon[76349]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:50.679 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:50.422Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:50.679 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:50.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:50.679 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:50.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:52.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:52 smithi184 ceph-mon[100622]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:52.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:52 smithi184 ceph-mon[102844]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:52.259 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:12:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:12:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:52 smithi196 ceph-mon[76349]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[100622]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[102844]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:53.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:53.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:12:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:12:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:12:53.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:12:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:54 smithi196 ceph-mon[76349]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[100622]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:55 smithi184 ceph-mon[102844]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:12:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:55 smithi196 ceph-mon[76349]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:58 smithi184 ceph-mon[100622]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:58 smithi184 ceph-mon[102844]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:58 smithi196 ceph-mon[76349]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:59.825 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:13:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:00 smithi184 ceph-mon[100622]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:00 smithi184 ceph-mon[102844]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:00.354 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:13:00.354 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 89m - - 2024-09-20T21:13:00.354 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 89m - - 2024-09-20T21:13:00.354 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (83m) 7s ago 84m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (84m) 5s ago 84m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (77m) 7s ago 82m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (77m) 5s ago 88m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (78m) 7s ago 90m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (77m) 7s ago 90m 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (76m) 5s ago 89m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (76m) 7s ago 89m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (84m) 7s ago 84m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (84m) 5s ago 84m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (75m) 7s ago 88m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (74m) 7s ago 88m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (73m) 7s ago 87m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (73m) 7s ago 87m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (71m) 5s ago 86m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:13:00.355 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (71m) 5s ago 86m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (69m) 5s ago 86m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (69m) 5s ago 85m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (78m) 5s ago 85m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (68m) 7s ago 83m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (68m) 5s ago 83m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (68m) 7s ago 83m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:13:00.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (68m) 5s ago 83m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:13:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:00 smithi196 ceph-mon[76349]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:00.664 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:13:00.665 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:13:00.665 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:00.422Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:00.665 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:00.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:00.665 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:00.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:00.933 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:13:00.933 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:13:00.934 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:13:01.250 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:13:01.250 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:01.251 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:01.251 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:01.251 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:01.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:01.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[100622]: from='client.49476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:01.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1463381072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:01.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:01.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[102844]: from='client.49476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:01.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1463381072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:01 smithi196 ceph-mon[76349]: from='client.49476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1463381072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:13:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[102844]: from='client.59447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[102844]: from='client.49488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[102844]: from='client.59465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[102844]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3734128971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[100622]: from='client.59447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[100622]: from='client.49488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[100622]: from='client.59465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[100622]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:02.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3734128971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:02 smithi196 ceph-mon[76349]: from='client.59447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:02 smithi196 ceph-mon[76349]: from='client.49488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:02 smithi196 ceph-mon[76349]: from='client.59465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:02 smithi196 ceph-mon[76349]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:02.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3734128971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:04 smithi184 ceph-mon[100622]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:04 smithi184 ceph-mon[102844]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:04.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:03.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:04.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:03.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:04.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:03.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:04 smithi196 ceph-mon[76349]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:06 smithi184 ceph-mon[100622]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:06 smithi184 ceph-mon[102844]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:06.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:06 smithi196 ceph-mon[76349]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:08 smithi184 ceph-mon[102844]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:08 smithi184 ceph-mon[100622]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:08 smithi196 ceph-mon[76349]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:10 smithi184 ceph-mon[100622]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:10 smithi184 ceph-mon[102844]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:10 smithi196 ceph-mon[76349]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:10.423Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:10.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:10.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:11.311 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:11.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:12 smithi184 ceph-mon[102844]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:12 smithi184 ceph-mon[100622]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:13:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:12 smithi196 ceph-mon[76349]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:13 smithi184 ceph-mon[100622]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:13 smithi184 ceph-mon[102844]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:13 smithi196 ceph-mon[76349]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:13.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:13.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:13.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:15 smithi196 ceph-mon[76349]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[100622]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:15 smithi184 ceph-mon[102844]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:18 smithi184 ceph-mon[100622]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:18 smithi184 ceph-mon[102844]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:18 smithi196 ceph-mon[76349]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:20 smithi184 ceph-mon[100622]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:20 smithi184 ceph-mon[102844]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:20 smithi196 ceph-mon[76349]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:20.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:20.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:20.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:22 smithi184 ceph-mon[100622]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:22 smithi184 ceph-mon[102844]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:13:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:22 smithi196 ceph-mon[76349]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:24 smithi184 ceph-mon[100622]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:24 smithi184 ceph-mon[102844]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:23.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:23.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:23.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:24 smithi196 ceph-mon[76349]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:26.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:26 smithi196 ceph-mon[76349]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:26 smithi184 ceph-mon[100622]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:26 smithi184 ceph-mon[102844]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:28 smithi184 ceph-mon[100622]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:28 smithi184 ceph-mon[102844]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:28 smithi196 ceph-mon[76349]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:30 smithi196 ceph-mon[76349]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:30.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:30 smithi184 ceph-mon[100622]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:30.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:30 smithi184 ceph-mon[102844]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:30.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:30.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:30.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:31 smithi196 ceph-mon[76349]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:31 smithi184 ceph-mon[100622]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:31 smithi184 ceph-mon[102844]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:31.550 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 90m - - 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 90m - - 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (84m) 18s ago 85m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (85m) 17s ago 85m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (78m) 18s ago 83m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (78m) 17s ago 89m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (79m) 18s ago 91m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (77m) 18s ago 91m 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (76m) 17s ago 89m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:13:32.095 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (77m) 18s ago 89m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (85m) 18s ago 85m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (85m) 17s ago 85m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (75m) 18s ago 89m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (74m) 18s ago 88m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (74m) 18s ago 88m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (73m) 18s ago 87m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (72m) 17s ago 87m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (71m) 17s ago 87m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (70m) 17s ago 86m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (69m) 17s ago 86m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (78m) 17s ago 85m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (69m) 18s ago 84m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (69m) 17s ago 84m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (69m) 18s ago 83m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:13:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (69m) 17s ago 83m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:13:32.146 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:32 smithi184 ceph-mon[100622]: from='client.59477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.146 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:32 smithi184 ceph-mon[100622]: from='client.59483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.147 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:13:32.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:32 smithi184 ceph-mon[102844]: from='client.59477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:32 smithi184 ceph-mon[102844]: from='client.59483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:32 smithi196 ceph-mon[76349]: from='client.59477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:32 smithi196 ceph-mon[76349]: from='client.59483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:32.417 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:13:32.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:13:32.419 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:13:32.699 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:13:32.699 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:13:32.700 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:13:33.039 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:13:33.039 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:33.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:33.040 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:33.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[102844]: from='client.49524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1755534704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[102844]: from='client.59501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[102844]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1558382712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[100622]: from='client.49524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1755534704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[100622]: from='client.59501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[100622]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1558382712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:33 smithi196 ceph-mon[76349]: from='client.49524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1755534704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:33 smithi196 ceph-mon[76349]: from='client.59501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:33 smithi196 ceph-mon[76349]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1558382712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:34.134 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:33.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:34.134 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:33.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:34.134 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:33.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[102844]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[100622]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:35 smithi196 ceph-mon[76349]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:38 smithi184 ceph-mon[100622]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:38 smithi184 ceph-mon[102844]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:38 smithi196 ceph-mon[76349]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:40 smithi184 ceph-mon[100622]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:40 smithi184 ceph-mon[102844]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:40 smithi196 ceph-mon[76349]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:40.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:40.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:40.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:42 smithi184 ceph-mon[100622]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:13:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:42 smithi184 ceph-mon[102844]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:42 smithi196 ceph-mon[76349]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:44 smithi184 ceph-mon[102844]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:44 smithi184 ceph-mon[100622]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:43.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:43.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:43.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:44 smithi196 ceph-mon[76349]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:46 smithi184 ceph-mon[100622]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:46 smithi184 ceph-mon[102844]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:46 smithi196 ceph-mon[76349]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:48 smithi184 ceph-mon[100622]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:48.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:48 smithi184 ceph-mon[102844]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:48.355 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:48 smithi196 ceph-mon[76349]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:50 smithi184 ceph-mon[100622]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:50 smithi184 ceph-mon[102844]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:50 smithi196 ceph-mon[76349]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:50.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:50.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:50.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:52 smithi184 ceph-mon[100622]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:52 smithi184 ceph-mon[102844]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:13:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:13:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:52 smithi196 ceph-mon[76349]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[100622]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[102844]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:53.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:53.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:13:54.302 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:13:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:13:53.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:13:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:54 smithi196 ceph-mon[76349]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:54.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[100622]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[102844]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:56 smithi196 ceph-mon[76349]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:13:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:58 smithi184 ceph-mon[100622]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:58 smithi184 ceph-mon[102844]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:58 smithi196 ceph-mon[76349]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:00 smithi196 ceph-mon[76349]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:00.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:00 smithi184 ceph-mon[100622]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:00.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:00 smithi184 ceph-mon[102844]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:00.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:00.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:00.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:02 smithi184 ceph-mon[100622]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:02 smithi184 ceph-mon[102844]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T21:14:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:02 smithi196 ceph-mon[76349]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:03 smithi184 ceph-mon[100622]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.340 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:03 smithi184 ceph-mon[102844]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.340 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:14:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:03 smithi196 ceph-mon[76349]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.875 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 90m - - 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 90m - - 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (84m) 10s ago 86m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (85m) 8s ago 85m 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (78m) 10s ago 84m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (78m) 8s ago 89m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (79m) 10s ago 91m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (78m) 10s ago 91m 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (77m) 8s ago 90m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (77m) 10s ago 90m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (85m) 10s ago 85m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (85m) 8s ago 85m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (76m) 10s ago 89m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (75m) 10s ago 89m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:14:03.876 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (74m) 10s ago 88m 63.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (74m) 10s ago 88m 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (72m) 8s ago 88m 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (72m) 8s ago 87m 64.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (71m) 8s ago 87m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (70m) 8s ago 86m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (79m) 8s ago 86m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (70m) 10s ago 84m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (69m) 8s ago 84m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (69m) 10s ago 84m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:14:03.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (69m) 8s ago 84m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:14:04.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:14:04.190 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:14:04.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[100622]: from='client.49548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[100622]: from='client.59519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[100622]: from='client.59525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[102844]: from='client.49548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[102844]: from='client.59519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:04 smithi184 ceph-mon[102844]: from='client.59525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.191 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:03.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:04.191 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:03.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:04.191 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:03.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:04 smithi196 ceph-mon[76349]: from='client.49548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:04 smithi196 ceph-mon[76349]: from='client.59519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:04 smithi196 ceph-mon[76349]: from='client.59525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:14:04.459 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:14:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:14:04.769 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:14:04.769 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:04.769 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/339355307' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:05 smithi196 ceph-mon[76349]: from='client.49572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3056792573' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:05 smithi196 ceph-mon[76349]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/339355307' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[100622]: from='client.49572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3056792573' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[100622]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/339355307' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[102844]: from='client.49572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3056792573' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:05 smithi184 ceph-mon[102844]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:14:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:08 smithi184 ceph-mon[100622]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:14:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:08 smithi184 ceph-mon[102844]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:14:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:08 smithi196 ceph-mon[76349]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:14:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:10 smithi184 ceph-mon[100622]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:14:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:10 smithi184 ceph-mon[102844]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:14:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:10 smithi196 ceph-mon[76349]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:14:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:10.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:10.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:10.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:12 smithi184 ceph-mon[100622]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:14:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:12 smithi184 ceph-mon[102844]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:14:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T21:14:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:12 smithi196 ceph-mon[76349]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:14:14.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[100622]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[102844]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:14.278 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:13.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:14.278 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:13.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:14.278 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:13.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:14 smithi196 ceph-mon[76349]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:14:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:16 smithi196 ceph-mon[76349]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:14:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[100622]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:14:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[102844]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:14:16.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:16.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:17 smithi196 ceph-mon[76349]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:14:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:17 smithi184 ceph-mon[100622]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:14:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:17 smithi184 ceph-mon[102844]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:14:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:20 smithi184 ceph-mon[100622]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:14:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:20 smithi184 ceph-mon[102844]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:14:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:20 smithi196 ceph-mon[76349]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:14:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:20.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:20.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:20.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:22 smithi184 ceph-mon[100622]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-20T21:14:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:22 smithi184 ceph-mon[102844]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-20T21:14:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:14:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:22 smithi196 ceph-mon[76349]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-20T21:14:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:24 smithi184 ceph-mon[100622]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T21:14:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:24 smithi184 ceph-mon[102844]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T21:14:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:23.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:23.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:23.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:24 smithi196 ceph-mon[76349]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T21:14:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:25 smithi196 ceph-mon[76349]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:14:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:25 smithi184 ceph-mon[100622]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:14:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:25 smithi184 ceph-mon[102844]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:14:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:27 smithi184 ceph-mon[100622]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:27 smithi184 ceph-mon[102844]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:27 smithi196 ceph-mon[76349]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:29.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:29.283 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:29.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:29 smithi196 ceph-mon[76349]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:14:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:29 smithi184 ceph-mon[100622]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:14:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:29 smithi184 ceph-mon[102844]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:14:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:30.428Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:30.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:30.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:31.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:31 smithi196 ceph-mon[76349]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:14:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:31 smithi184 ceph-mon[100622]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:14:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:14:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:31 smithi184 ceph-mon[102844]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:14:34.140 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[100622]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[102844]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.141 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:33.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:34.142 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:33.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:34.142 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:33.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:34 smithi196 ceph-mon[76349]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:14:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 91m - - 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 91m - - 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (85m) 1s ago 86m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (86m) 0s ago 86m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (79m) 1s ago 84m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (79m) 0s ago 90m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:14:35.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (80m) 1s ago 92m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (78m) 1s ago 92m 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (77m) 0s ago 90m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (78m) 1s ago 90m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (86m) 1s ago 86m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (86m) 0s ago 86m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (76m) 1s ago 90m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (75m) 1s ago 89m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (75m) 1s ago 89m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (74m) 1s ago 88m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (73m) 0s ago 88m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (72m) 0s ago 88m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (71m) 0s ago 87m 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (70m) 0s ago 87m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:14:35.614 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (79m) 0s ago 86m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:14:35.615 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (70m) 1s ago 85m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:14:35.615 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (70m) 0s ago 85m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:14:35.615 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (70m) 1s ago 84m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:14:35.615 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (70m) 0s ago 84m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:14:35.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:14:35.933 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:14:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:14:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:14:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:14:36.210 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: from='client.49584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3821441722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: from='client.49584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3821441722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-20T21:14:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: from='client.49584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3821441722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:36.521 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:14:36.522 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:36.522 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:36.522 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:36.522 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:37 smithi196 ceph-mon[76349]: from='client.59555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:37 smithi196 ceph-mon[76349]: from='client.49596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2430325505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[100622]: from='client.59555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[100622]: from='client.49596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2430325505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[102844]: from='client.59555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[102844]: from='client.49596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2430325505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:38 smithi196 ceph-mon[76349]: from='client.59573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:38 smithi196 ceph-mon[76349]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:14:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:38 smithi184 ceph-mon[100622]: from='client.59573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:38 smithi184 ceph-mon[100622]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:14:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:38 smithi184 ceph-mon[102844]: from='client.59573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:38 smithi184 ceph-mon[102844]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:14:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:39 smithi196 ceph-mon[76349]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:14:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:39 smithi184 ceph-mon[100622]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:14:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:39 smithi184 ceph-mon[102844]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:14:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:40.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:40.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:40.428Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:40.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:40.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:41 smithi196 ceph-mon[76349]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:14:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:41 smithi184 ceph-mon[100622]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:14:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:41 smithi184 ceph-mon[102844]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:14:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:14:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:44 smithi184 ceph-mon[100622]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:44 smithi184 ceph-mon[102844]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:43.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:43.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:44.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:43.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:44 smithi196 ceph-mon[76349]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:46 smithi196 ceph-mon[76349]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:46 smithi184 ceph-mon[100622]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:46 smithi184 ceph-mon[102844]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:48 smithi196 ceph-mon[76349]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:48 smithi184 ceph-mon[100622]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:48 smithi184 ceph-mon[102844]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:50 smithi196 ceph-mon[76349]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:50.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:50 smithi184 ceph-mon[100622]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:50.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:50 smithi184 ceph-mon[102844]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:50.429Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:50.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:50.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:51 smithi196 ceph-mon[76349]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:51 smithi184 ceph-mon[100622]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:51 smithi184 ceph-mon[102844]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:14:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[100622]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[102844]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:53.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:53.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:14:54.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:14:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:14:53.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:14:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:54 smithi196 ceph-mon[76349]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[100622]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[102844]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:56 smithi196 ceph-mon[76349]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:14:57.091 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:57 smithi184 ceph-mon[100622]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:57.092 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:57 smithi184 ceph-mon[102844]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:57 smithi196 ceph-mon[76349]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:14:59 smithi196 ceph-mon[76349]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:14:59 smithi184 ceph-mon[100622]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:14:59 smithi184 ceph-mon[102844]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:00.430Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:00.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:00.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:01 smithi196 ceph-mon[76349]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:01 smithi184 ceph-mon[100622]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:01 smithi184 ceph-mon[102844]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T21:15:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:03.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:03.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:03.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:04.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:04 smithi184 ceph-mon[100622]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:04.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:04 smithi184 ceph-mon[102844]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:04 smithi196 ceph-mon[76349]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:06 smithi196 ceph-mon[76349]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:06 smithi184 ceph-mon[100622]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:06 smithi184 ceph-mon[102844]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:06.818 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 91m - - 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 91m - - 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (86m) 13s ago 87m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (86m) 11s ago 86m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (80m) 13s ago 85m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (79m) 11s ago 90m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (80m) 13s ago 92m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (79m) 13s ago 92m 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:15:07.348 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (78m) 11s ago 91m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (78m) 13s ago 91m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (86m) 13s ago 86m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (86m) 11s ago 86m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (77m) 13s ago 90m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (76m) 13s ago 90m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (75m) 13s ago 89m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (75m) 13s ago 89m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (74m) 11s ago 89m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (73m) 11s ago 88m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (72m) 11s ago 88m 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (71m) 11s ago 87m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (80m) 11s ago 87m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (71m) 13s ago 85m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (70m) 11s ago 85m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:15:07.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (70m) 13s ago 85m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:15:07.350 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (70m) 11s ago 85m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:07 smithi196 ceph-mon[76349]: from='client.49620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:07 smithi196 ceph-mon[76349]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:07 smithi184 ceph-mon[100622]: from='client.49620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:07 smithi184 ceph-mon[100622]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:07 smithi184 ceph-mon[102844]: from='client.49620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:07 smithi184 ceph-mon[102844]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:15:07.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:07.663 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:15:07.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:15:07.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:15:07.663 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:15:07.933 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:15:07.933 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:15:07.933 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:15:07.934 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:15:08.248 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:15:08.248 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:08.248 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:08.248 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[100622]: from='client.59591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[100622]: from='client.59597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3872736435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[100622]: from='client.49644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[102844]: from='client.59591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[102844]: from='client.59597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3872736435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:08.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:08 smithi184 ceph-mon[102844]: from='client.49644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:08 smithi196 ceph-mon[76349]: from='client.59591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:08 smithi196 ceph-mon[76349]: from='client.59597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3872736435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:08 smithi196 ceph-mon[76349]: from='client.49644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:09.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1253738941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:09.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:09 smithi196 ceph-mon[76349]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1253738941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:09 smithi184 ceph-mon[100622]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1253738941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:09 smithi184 ceph-mon[102844]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:10.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:10.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:10.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:10.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:10.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:11.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:11 smithi196 ceph-mon[76349]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:11 smithi184 ceph-mon[100622]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:11 smithi184 ceph-mon[102844]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-20T21:15:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:14 smithi184 ceph-mon[100622]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:14 smithi184 ceph-mon[102844]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:13.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:13.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:13.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:14 smithi196 ceph-mon[76349]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:15.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.347 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[100622]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:15.348 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.348 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.348 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.348 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:15 smithi184 ceph-mon[102844]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:15 smithi196 ceph-mon[76349]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:17 smithi184 ceph-mon[100622]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:17 smithi184 ceph-mon[102844]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:17 smithi196 ceph-mon[76349]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:20 smithi196 ceph-mon[76349]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:20.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:20 smithi184 ceph-mon[100622]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:20.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:20 smithi184 ceph-mon[102844]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:20.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:20.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:20.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:21 smithi196 ceph-mon[76349]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:21 smithi184 ceph-mon[100622]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:21 smithi184 ceph-mon[102844]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:15:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:23 smithi196 ceph-mon[76349]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:23 smithi184 ceph-mon[100622]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:23 smithi184 ceph-mon[102844]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:23.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:23.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:23.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:25 smithi196 ceph-mon[76349]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:25 smithi184 ceph-mon[100622]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:25 smithi184 ceph-mon[102844]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:27.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:27 smithi184 ceph-mon[100622]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:27.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:27 smithi184 ceph-mon[102844]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:27 smithi196 ceph-mon[76349]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:30.118 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:29 smithi196 ceph-mon[76349]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:29 smithi184 ceph-mon[100622]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:29 smithi184 ceph-mon[102844]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:30.432Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:30.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:30.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:31.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:31 smithi196 ceph-mon[76349]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:31 smithi184 ceph-mon[100622]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:31 smithi184 ceph-mon[102844]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:15:33.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:33 smithi184 ceph-mon[100622]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:33.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:33 smithi184 ceph-mon[102844]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:33 smithi196 ceph-mon[76349]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:34.236 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:33.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:34.237 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:33.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:34.237 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:33.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[102844]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:35 smithi184 ceph-mon[100622]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:35 smithi196 ceph-mon[76349]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:37 smithi184 ceph-mon[100622]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:37 smithi184 ceph-mon[102844]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:37 smithi196 ceph-mon[76349]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:38.538 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:15:39.064 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:15:39.064 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 92m - - 2024-09-20T21:15:39.064 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 92m - - 2024-09-20T21:15:39.064 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (86m) 4s ago 87m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (87m) 3s ago 87m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (80m) 4s ago 85m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (80m) 3s ago 91m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (81m) 4s ago 93m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (79m) 4s ago 93m 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (79m) 3s ago 91m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (79m) 4s ago 91m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (87m) 4s ago 87m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (87m) 3s ago 87m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (77m) 4s ago 91m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (77m) 4s ago 90m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (76m) 4s ago 90m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (75m) 4s ago 90m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (74m) 3s ago 89m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:15:39.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (73m) 3s ago 89m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (72m) 3s ago 88m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (71m) 3s ago 88m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (80m) 3s ago 88m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (71m) 4s ago 86m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (71m) 3s ago 86m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (71m) 4s ago 85m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:15:39.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (71m) 3s ago 85m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[100622]: from='client.59621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[100622]: from='client.49662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[100622]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[100622]: from='client.49668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[102844]: from='client.59621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[102844]: from='client.49662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[102844]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:39.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:39 smithi184 ceph-mon[102844]: from='client.49668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:15:39.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:15:39.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:15:39.382 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:15:39.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:15:39.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:15:39.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:15:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:39 smithi196 ceph-mon[76349]: from='client.59621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:39 smithi196 ceph-mon[76349]: from='client.49662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:39 smithi196 ceph-mon[76349]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:39 smithi196 ceph-mon[76349]: from='client.49668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:15:39.651 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:15:39.652 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:15:39.652 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:15:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:15:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3831698969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[100622]: from='client.59645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2971969526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3831698969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[102844]: from='client.59645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2971969526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3831698969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:40 smithi196 ceph-mon[76349]: from='client.59645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2971969526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:40.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:40.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:40.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:41 smithi196 ceph-mon[76349]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:41 smithi184 ceph-mon[100622]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:41 smithi184 ceph-mon[102844]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:15:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:43 smithi184 ceph-mon[100622]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:43 smithi184 ceph-mon[102844]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:43 smithi196 ceph-mon[76349]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:43.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:43.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:43.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:45 smithi196 ceph-mon[76349]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:45 smithi184 ceph-mon[100622]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:45 smithi184 ceph-mon[102844]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:47 smithi196 ceph-mon[76349]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:47 smithi184 ceph-mon[100622]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:47 smithi184 ceph-mon[102844]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:49.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:49 smithi184 ceph-mon[100622]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:49 smithi184 ceph-mon[102844]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:49 smithi196 ceph-mon[76349]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:50.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:50.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:50.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:50.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:50.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:50.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:51 smithi196 ceph-mon[76349]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:51 smithi184 ceph-mon[100622]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:51 smithi184 ceph-mon[102844]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:15:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-20T21:15:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:53 smithi196 ceph-mon[76349]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:53 smithi184 ceph-mon[100622]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:53 smithi184 ceph-mon[102844]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:53.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:53.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:15:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:15:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:15:53.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:15:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:55 smithi196 ceph-mon[76349]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[100622]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:55 smithi184 ceph-mon[102844]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:57.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:15:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:57 smithi184 ceph-mon[100622]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:57 smithi184 ceph-mon[102844]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:57 smithi196 ceph-mon[76349]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:59.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:15:59 smithi184 ceph-mon[100622]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:15:59 smithi184 ceph-mon[102844]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:00.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:15:59 smithi196 ceph-mon[76349]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:00.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:00.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:00.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:01 smithi196 ceph-mon[76349]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:01 smithi184 ceph-mon[100622]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:02.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:16:02.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:01 smithi184 ceph-mon[102844]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:03 smithi196 ceph-mon[76349]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:03 smithi184 ceph-mon[100622]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:03 smithi184 ceph-mon[102844]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:03.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:03.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:03.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:05 smithi196 ceph-mon[76349]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:05 smithi184 ceph-mon[100622]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:05 smithi184 ceph-mon[102844]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:07 smithi196 ceph-mon[76349]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:07 smithi184 ceph-mon[100622]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:07 smithi184 ceph-mon[102844]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:09 smithi196 ceph-mon[76349]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:09 smithi184 ceph-mon[100622]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:09 smithi184 ceph-mon[102844]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:10.251 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:16:10.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:10.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:10.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 92m - - 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 92m - - 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (87m) 16s ago 88m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (87m) 14s ago 87m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (81m) 16s ago 86m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (80m) 14s ago 92m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:16:10.775 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (81m) 16s ago 93m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (80m) 16s ago 93m 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (79m) 14s ago 92m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (79m) 16s ago 92m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (87m) 16s ago 87m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (87m) 14s ago 87m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (78m) 16s ago 91m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (77m) 16s ago 91m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (76m) 16s ago 90m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (76m) 16s ago 90m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (75m) 14s ago 90m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (74m) 14s ago 89m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (73m) 14s ago 89m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (72m) 14s ago 89m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (81m) 14s ago 88m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (72m) 16s ago 86m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:16:10.776 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (71m) 14s ago 86m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:16:10.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (72m) 16s ago 86m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:16:10.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (71m) 14s ago 86m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:16:10.777 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:10.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:10.777 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:10.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:10.777 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:10.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:16:11.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:16:11.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:11.086 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:16:11.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:16:11.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:16:11.086 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:16:11.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:11 smithi196 ceph-mon[76349]: from='client.49692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:11 smithi196 ceph-mon[76349]: from='client.59663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:11 smithi196 ceph-mon[76349]: from='client.59669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:11 smithi196 ceph-mon[76349]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:11.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2032474262' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:16:11.354 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:16:11.355 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:16:11.355 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[100622]: from='client.49692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[100622]: from='client.59663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[100622]: from='client.59669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[100622]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2032474262' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[102844]: from='client.49692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[102844]: from='client.59663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.355 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[102844]: from='client.59669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:11.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[102844]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:11.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2032474262' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:11.663 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:16:11.663 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:11.663 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:12.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:12 smithi196 ceph-mon[76349]: from='client.59681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:12.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/527580310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:12.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:12 smithi184 ceph-mon[100622]: from='client.59681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:12.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/527580310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:12.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:12 smithi184 ceph-mon[102844]: from='client.59681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:12.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/527580310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:12.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:16:13.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:13 smithi196 ceph-mon[76349]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:13 smithi184 ceph-mon[100622]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:13 smithi184 ceph-mon[102844]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:14.138 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:13.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:14.138 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:13.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:14.138 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:13.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:14.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:14.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[100622]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:15 smithi184 ceph-mon[102844]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:15 smithi196 ceph-mon[76349]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:18 smithi184 ceph-mon[100622]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:18 smithi184 ceph-mon[102844]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:18 smithi196 ceph-mon[76349]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:19.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:19 smithi184 ceph-mon[100622]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:19.372 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:19 smithi184 ceph-mon[102844]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:19 smithi196 ceph-mon[76349]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:20.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:20.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:20.436Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:20.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:20.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:21 smithi196 ceph-mon[76349]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:21 smithi184 ceph-mon[100622]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:21 smithi184 ceph-mon[102844]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:23 smithi196 ceph-mon[76349]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:23 smithi184 ceph-mon[100622]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:23 smithi184 ceph-mon[102844]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:23.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:23.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:23.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:25 smithi196 ceph-mon[76349]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:25 smithi184 ceph-mon[100622]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:25 smithi184 ceph-mon[102844]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:27 smithi196 ceph-mon[76349]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:27 smithi184 ceph-mon[100622]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:27 smithi184 ceph-mon[102844]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:29 smithi196 ceph-mon[76349]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:29 smithi184 ceph-mon[100622]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:29 smithi184 ceph-mon[102844]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:30.436Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:30.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:30.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:31 smithi196 ceph-mon[76349]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:31 smithi184 ceph-mon[100622]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:31 smithi184 ceph-mon[102844]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:16:33.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:33 smithi196 ceph-mon[76349]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:33.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:33 smithi184 ceph-mon[100622]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:33.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:33 smithi184 ceph-mon[102844]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:33.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:33.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:33.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:35 smithi196 ceph-mon[76349]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[100622]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:36.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:35 smithi184 ceph-mon[102844]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[100622]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:37 smithi184 ceph-mon[102844]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:37.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:37 smithi196 ceph-mon[76349]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:39.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:39 smithi184 ceph-mon[100622]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:39.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:39 smithi184 ceph-mon[102844]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:39 smithi196 ceph-mon[76349]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:40.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:40.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:40.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:40.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:40.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:41 smithi196 ceph-mon[76349]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:41 smithi184 ceph-mon[100622]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:41 smithi184 ceph-mon[102844]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:41.957 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:16:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:42 smithi184 ceph-mon[100622]: from='client.49728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:42 smithi184 ceph-mon[102844]: from='client.49728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:16:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:42 smithi196 ceph-mon[76349]: from='client.49728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 93m - - 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 93m - - 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (87m) 7s ago 88m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (88m) 6s ago 88m 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (81m) 7s ago 86m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (81m) 6s ago 92m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (82m) 7s ago 94m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (80m) 7s ago 94m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (80m) 6s ago 93m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:16:42.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (80m) 7s ago 92m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (88m) 7s ago 88m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (88m) 6s ago 88m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (78m) 7s ago 92m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (78m) 7s ago 91m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (77m) 7s ago 91m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (76m) 7s ago 91m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (75m) 6s ago 90m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (74m) 6s ago 90m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (73m) 6s ago 89m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (73m) 6s ago 89m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (81m) 6s ago 89m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (72m) 7s ago 87m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:16:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (72m) 6s ago 87m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:16:42.486 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (72m) 7s ago 86m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:16:42.486 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (72m) 6s ago 86m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:16:42.794 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:16:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:16:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:16:42.795 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:16:43.065 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:16:43.066 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:16:43.066 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:16:43.378 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:16:43.378 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:43.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:43.378 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:43.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:43.378 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[102844]: from='client.49734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[102844]: from='client.49740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3278166848' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[102844]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[102844]: from='client.59717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[100622]: from='client.49734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[100622]: from='client.49740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3278166848' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[100622]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:43.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-mon[100622]: from='client.59717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:43 smithi196 ceph-mon[76349]: from='client.49734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:43 smithi196 ceph-mon[76349]: from='client.49740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3278166848' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:43 smithi196 ceph-mon[76349]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:43 smithi196 ceph-mon[76349]: from='client.59717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:44.147 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:43.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:44.147 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:43.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:44.148 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:43.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/518125696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/518125696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/518125696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:45 smithi184 ceph-mon[100622]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:45 smithi184 ceph-mon[102844]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:45 smithi196 ceph-mon[76349]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:47 smithi196 ceph-mon[76349]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:47 smithi184 ceph-mon[100622]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:47 smithi184 ceph-mon[102844]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:49 smithi196 ceph-mon[76349]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:49.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:49 smithi184 ceph-mon[100622]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:49.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:49 smithi184 ceph-mon[102844]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:50.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:50.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:50.438Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:50.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:50.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:51 smithi196 ceph-mon[76349]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:51 smithi184 ceph-mon[100622]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:51 smithi184 ceph-mon[102844]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:16:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:16:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:53 smithi196 ceph-mon[76349]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:53 smithi184 ceph-mon[100622]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:53 smithi184 ceph-mon[102844]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:53.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:53.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:16:54.302 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:16:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:16:53.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[100622]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:55 smithi184 ceph-mon[102844]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:56.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:55 smithi196 ceph-mon[76349]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[100622]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:57 smithi184 ceph-mon[102844]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:57 smithi196 ceph-mon[76349]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:16:59 smithi196 ceph-mon[76349]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:59.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:16:59 smithi184 ceph-mon[100622]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:59.930 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:16:59 smithi184 ceph-mon[102844]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:00.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:00.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:00.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:01.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:01.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:01 smithi196 ceph-mon[76349]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:01 smithi184 ceph-mon[100622]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:01 smithi184 ceph-mon[102844]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:17:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:03 smithi196 ceph-mon[76349]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:03.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:03 smithi184 ceph-mon[100622]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:03.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:03 smithi184 ceph-mon[102844]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:03.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:03.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:03.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:05 smithi196 ceph-mon[76349]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:05 smithi184 ceph-mon[100622]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:05 smithi184 ceph-mon[102844]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:07 smithi196 ceph-mon[76349]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:07 smithi184 ceph-mon[100622]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:07 smithi184 ceph-mon[102844]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:09 smithi196 ceph-mon[76349]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:09.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:09 smithi184 ceph-mon[100622]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:09.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:09 smithi184 ceph-mon[102844]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:10.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:10.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:10.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:11 smithi184 ceph-mon[100622]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:11 smithi184 ceph-mon[102844]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:11 smithi196 ceph-mon[76349]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:17:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:13 smithi196 ceph-mon[76349]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:13 smithi184 ceph-mon[100622]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:13 smithi184 ceph-mon[102844]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.676 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:17:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:13.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:13.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:13.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 93m - - 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 93m - - 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (88m) 19s ago 89m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (89m) 17s ago 89m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (82m) 19s ago 87m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (81m) 17s ago 93m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (82m) 19s ago 94m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:17:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (81m) 19s ago 94m 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (80m) 17s ago 93m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (80m) 19s ago 93m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (88m) 19s ago 88m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (88m) 17s ago 88m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (79m) 19s ago 92m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (78m) 19s ago 92m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (77m) 19s ago 91m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (77m) 19s ago 91m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (76m) 17s ago 91m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (75m) 17s ago 90m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (74m) 17s ago 90m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (73m) 17s ago 90m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (82m) 17s ago 89m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (73m) 19s ago 87m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (72m) 17s ago 87m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:17:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (73m) 19s ago 87m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:17:14.210 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (72m) 17s ago 87m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:17:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:14 smithi196 ceph-mon[76349]: from='client.49764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:14 smithi196 ceph-mon[76349]: from='client.49770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[100622]: from='client.49764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:14.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[100622]: from='client.49770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[102844]: from='client.49764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:14.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:14 smithi184 ceph-mon[102844]: from='client.49770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:14.543 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:17:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:17:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:17:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:17:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:17:14.819 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:17:14.820 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:17:14.820 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:17:14.820 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:17:14.820 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:17:15.132 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:17:15.132 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:15.132 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:15.132 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:15.132 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='client.49776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1812122343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='client.49788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:15.224 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/600859654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='client.49776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1812122343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='client.49788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/600859654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='client.49776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1812122343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='client.49788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:15.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/600859654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[100622]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:17 smithi184 ceph-mon[102844]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:17 smithi196 ceph-mon[76349]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:19.385 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:19 smithi184 ceph-mon[100622]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:19.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:19 smithi184 ceph-mon[102844]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:19 smithi196 ceph-mon[76349]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:20.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:20.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:20.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:20.441Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:20.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:20.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:20.693 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:20.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:21.325 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:21 smithi184 ceph-mon[100622]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:21.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:21 smithi184 ceph-mon[102844]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:21 smithi196 ceph-mon[76349]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:22.265 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:17:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:23 smithi196 ceph-mon[76349]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:23 smithi184 ceph-mon[100622]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:23 smithi184 ceph-mon[102844]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:23.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:23.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:23.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:25 smithi184 ceph-mon[100622]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:25 smithi184 ceph-mon[102844]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:25 smithi196 ceph-mon[76349]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:27 smithi184 ceph-mon[100622]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:27 smithi184 ceph-mon[102844]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:27 smithi196 ceph-mon[76349]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:29 smithi196 ceph-mon[76349]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:30.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:29 smithi184 ceph-mon[100622]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:30.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:29 smithi184 ceph-mon[102844]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:30.442Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:30.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:30.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:32.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:31 smithi184 ceph-mon[102844]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:31 smithi184 ceph-mon[100622]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:17:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:31 smithi196 ceph-mon[76349]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:33 smithi196 ceph-mon[76349]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:33 smithi184 ceph-mon[100622]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:33 smithi184 ceph-mon[102844]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:33.880Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:33.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:33.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[100622]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:35 smithi184 ceph-mon[102844]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:35 smithi196 ceph-mon[76349]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[100622]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:37 smithi184 ceph-mon[102844]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:37 smithi196 ceph-mon[76349]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:39.445 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:39 smithi184 ceph-mon[100622]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:39.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:39 smithi184 ceph-mon[102844]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:39 smithi196 ceph-mon[76349]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:40.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:40.445 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:40.443Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:40.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:40.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:41.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:41 smithi184 ceph-mon[100622]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:41.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:41 smithi184 ceph-mon[102844]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:41 smithi196 ceph-mon[76349]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:17:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:43 smithi184 ceph-mon[100622]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:43 smithi184 ceph-mon[102844]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:43 smithi196 ceph-mon[76349]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:44.142 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:43.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:44.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:43.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:44.143 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:43.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:44.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:44.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:45.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:45 smithi184 ceph-mon[100622]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:45.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:45 smithi184 ceph-mon[102844]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:45.434 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:17:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:45 smithi196 ceph-mon[76349]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 94m - - 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 94m - - 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (88m) 10s ago 89m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (89m) 9s ago 89m 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (82m) 10s ago 87m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:17:45.962 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (82m) 9s ago 93m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (83m) 10s ago 95m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (81m) 10s ago 95m 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (81m) 9s ago 94m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (81m) 10s ago 93m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (89m) 10s ago 89m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (89m) 9s ago 89m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (80m) 10s ago 93m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:17:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (79m) 10s ago 92m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (78m) 10s ago 92m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (77m) 10s ago 92m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (76m) 9s ago 91m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (75m) 9s ago 91m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (74m) 9s ago 91m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (74m) 9s ago 90m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (82m) 9s ago 90m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (73m) 10s ago 88m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (73m) 9s ago 88m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (73m) 10s ago 87m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:17:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (73m) 9s ago 87m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:17:46.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[100622]: from='client.49800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[100622]: from='client.59771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[100622]: from='client.49812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:17:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:17:46.273 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:17:46.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[102844]: from='client.49800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[102844]: from='client.59771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:46 smithi184 ceph-mon[102844]: from='client.49812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.540 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:17:46.540 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:17:46.541 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:17:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:46 smithi196 ceph-mon[76349]: from='client.49800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:46 smithi196 ceph-mon[76349]: from='client.59771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:46 smithi196 ceph-mon[76349]: from='client.49812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:46.855 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:17:46.856 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:46.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:46.856 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:46.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3480227497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[100622]: from='client.49824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3754785904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[100622]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3480227497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[102844]: from='client.49824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3754785904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:47 smithi184 ceph-mon[102844]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3480227497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:47 smithi196 ceph-mon[76349]: from='client.49824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3754785904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:47 smithi196 ceph-mon[76349]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:49.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:49 smithi184 ceph-mon[100622]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:49.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:49 smithi184 ceph-mon[102844]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:49 smithi196 ceph-mon[76349]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:50.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:50.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:50.443Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:50.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:50.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:51 smithi184 ceph-mon[100622]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:51 smithi184 ceph-mon[102844]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:51 smithi196 ceph-mon[76349]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:17:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:17:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:53 smithi184 ceph-mon[100622]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:53 smithi184 ceph-mon[102844]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:53 smithi196 ceph-mon[76349]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:53.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:53.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:17:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:17:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:17:53.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:17:55.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:55 smithi196 ceph-mon[76349]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:55.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[100622]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[102844]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:55.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[100622]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:57 smithi184 ceph-mon[102844]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:17:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:57 smithi196 ceph-mon[76349]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:17:59 smithi196 ceph-mon[76349]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:17:59 smithi184 ceph-mon[100622]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:17:59 smithi184 ceph-mon[102844]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:00.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:00.444Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:00.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:00.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:00.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:00.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:01.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:01 smithi184 ceph-mon[100622]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:01.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:01 smithi184 ceph-mon[102844]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:01 smithi196 ceph-mon[76349]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:03 smithi184 ceph-mon[100622]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:03 smithi184 ceph-mon[102844]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:03 smithi196 ceph-mon[76349]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:03.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:03.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:03.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:05 smithi184 ceph-mon[100622]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:05 smithi184 ceph-mon[102844]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:05 smithi196 ceph-mon[76349]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:07 smithi184 ceph-mon[100622]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:07 smithi184 ceph-mon[102844]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:07 smithi196 ceph-mon[76349]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:09.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:09 smithi184 ceph-mon[100622]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:09.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:09 smithi184 ceph-mon[102844]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:09 smithi196 ceph-mon[76349]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:10.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:10.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:10.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:10.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:10.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:11 smithi184 ceph-mon[100622]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:11 smithi184 ceph-mon[102844]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:11 smithi196 ceph-mon[76349]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:13 smithi184 ceph-mon[100622]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:13 smithi184 ceph-mon[102844]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:13 smithi196 ceph-mon[76349]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:13.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:13.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:13.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:18:13.812536762Z level=info msg="Completed cleanup jobs" duration=13.044121ms 2024-09-20T21:18:14.463 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:15 smithi184 ceph-mon[100622]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:15 smithi184 ceph-mon[102844]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:15.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:15 smithi196 ceph-mon[76349]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:16.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.149 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:18:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:18:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 95m - - 2024-09-20T21:18:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 94m - - 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (89m) 2s ago 90m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (90m) 0s ago 90m 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (83m) 2s ago 88m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (83m) 0s ago 94m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (83m) 2s ago 95m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (82m) 2s ago 95m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (81m) 0s ago 94m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (82m) 2s ago 94m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (89m) 2s ago 89m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (89m) 0s ago 89m 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (80m) 2s ago 93m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (79m) 2s ago 93m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (79m) 2s ago 93m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:18:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (78m) 2s ago 92m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (77m) 0s ago 92m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (76m) 0s ago 91m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (75m) 0s ago 91m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (74m) 0s ago 91m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (83m) 0s ago 90m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (74m) 2s ago 88m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (74m) 0s ago 88m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (74m) 2s ago 88m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:18:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (73m) 0s ago 88m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:18:17.995 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:18:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:18:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:18:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:18:17.998 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:18:17.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[100622]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:17.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:17.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:18.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:17 smithi184 ceph-mon[102844]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:17 smithi196 ceph-mon[76349]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:18:18.265 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:18:18.266 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:18:18.266 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:18:18.266 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:18:18.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:18:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:18:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:18.578 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:18.578 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[100622]: from='client.49836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[100622]: from='client.49842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[100622]: from='client.59813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/815373520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1870713226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[102844]: from='client.49836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[102844]: from='client.49842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[102844]: from='client.59813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/815373520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1870713226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:18 smithi196 ceph-mon[76349]: from='client.49836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:18 smithi196 ceph-mon[76349]: from='client.49842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:18 smithi196 ceph-mon[76349]: from='client.59813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/815373520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1870713226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:19 smithi196 ceph-mon[76349]: from='client.49860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:19 smithi196 ceph-mon[76349]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:19 smithi184 ceph-mon[100622]: from='client.49860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:19 smithi184 ceph-mon[100622]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:19 smithi184 ceph-mon[102844]: from='client.49860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:19 smithi184 ceph-mon[102844]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:20.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:20.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:20.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:21.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:21.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:21 smithi196 ceph-mon[76349]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:22.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:21 smithi184 ceph-mon[100622]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:22.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:21 smithi184 ceph-mon[102844]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:22.270 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:23 smithi184 ceph-mon[100622]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:23 smithi184 ceph-mon[102844]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:23 smithi196 ceph-mon[76349]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:23.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:23.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:23.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:25 smithi184 ceph-mon[100622]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:25 smithi184 ceph-mon[102844]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:25 smithi196 ceph-mon[76349]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:27 smithi184 ceph-mon[100622]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:27 smithi184 ceph-mon[102844]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:27 smithi196 ceph-mon[76349]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:29 smithi196 ceph-mon[76349]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:29 smithi184 ceph-mon[100622]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:29 smithi184 ceph-mon[102844]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:30.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:30.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:30.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:31 smithi196 ceph-mon[76349]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:31 smithi184 ceph-mon[100622]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:31 smithi184 ceph-mon[102844]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:33.442 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:33 smithi184 ceph-mon[100622]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:33.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:33 smithi184 ceph-mon[102844]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:33 smithi196 ceph-mon[76349]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:33.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:33.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:33.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:35.254 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:35 smithi196 ceph-mon[76349]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:35 smithi184 ceph-mon[100622]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:35 smithi184 ceph-mon[102844]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:37 smithi196 ceph-mon[76349]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[100622]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:37 smithi184 ceph-mon[102844]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:39.496 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:39 smithi184 ceph-mon[100622]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:39.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:39 smithi184 ceph-mon[102844]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:39 smithi196 ceph-mon[76349]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:40.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:40.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:40.447Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:40.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:40.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:41 smithi184 ceph-mon[100622]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:41 smithi184 ceph-mon[102844]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:41 smithi196 ceph-mon[76349]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:42.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:43.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:43 smithi184 ceph-mon[100622]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:43.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:43 smithi184 ceph-mon[102844]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:43 smithi196 ceph-mon[76349]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:44.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:43.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:44.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:43.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:44.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:43.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:44.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:44.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:45 smithi184 ceph-mon[100622]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:45 smithi184 ceph-mon[102844]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:45 smithi196 ceph-mon[76349]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:47 smithi184 ceph-mon[100622]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:47 smithi184 ceph-mon[102844]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:47 smithi196 ceph-mon[76349]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:48.872 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:18:49.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:49 smithi184 ceph-mon[100622]: from='client.49872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:49.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:49 smithi184 ceph-mon[100622]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:49 smithi184 ceph-mon[102844]: from='client.49872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:49 smithi184 ceph-mon[102844]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:49.403 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:18:49.403 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 95m - - 2024-09-20T21:18:49.403 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 95m - - 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (89m) 14s ago 90m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (90m) 12s ago 90m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (83m) 14s ago 88m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (83m) 12s ago 94m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (84m) 14s ago 96m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (82m) 14s ago 96m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (82m) 12s ago 95m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (82m) 14s ago 94m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (90m) 14s ago 90m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (90m) 12s ago 90m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (81m) 14s ago 94m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (80m) 14s ago 93m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (79m) 14s ago 93m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:18:49.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (79m) 14s ago 93m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (77m) 12s ago 92m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (76m) 12s ago 92m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (75m) 12s ago 92m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (75m) 12s ago 91m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (83m) 12s ago 91m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (74m) 14s ago 89m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (74m) 12s ago 89m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (74m) 14s ago 88m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:18:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (74m) 12s ago 88m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:18:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:49 smithi196 ceph-mon[76349]: from='client.49872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:49 smithi196 ceph-mon[76349]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:49.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:18:49.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:18:49.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:18:49.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:49.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:18:49.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:18:49.983 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:18:49.984 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:18:49.984 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:18:49.984 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:18:49.984 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:18:50.292 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:18:50.293 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:50.293 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:50.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[100622]: from='client.49878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[100622]: from='client.49884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:50.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3786389743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:50.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[100622]: from='client.59861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[102844]: from='client.49878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[102844]: from='client.49884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:50.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3786389743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:50.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-mon[102844]: from='client.59861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:50.447Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:50.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:50.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:50 smithi196 ceph-mon[76349]: from='client.49878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:50 smithi196 ceph-mon[76349]: from='client.49884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3786389743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:50 smithi196 ceph-mon[76349]: from='client.59861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/858600907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:51 smithi184 ceph-mon[100622]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/858600907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:51 smithi184 ceph-mon[102844]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/858600907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:51 smithi196 ceph-mon[76349]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:18:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:18:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:53 smithi184 ceph-mon[100622]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:53 smithi184 ceph-mon[102844]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:53 smithi196 ceph-mon[76349]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:53.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:53.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:18:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:18:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:18:53.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:18:55.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:55 smithi184 ceph-mon[100622]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:55.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:55 smithi184 ceph-mon[102844]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:55.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:55 smithi196 ceph-mon[76349]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:56.806 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:58 smithi196 ceph-mon[76349]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[100622]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:18:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:58 smithi184 ceph-mon[102844]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:59.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:59.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:00 smithi196 ceph-mon[76349]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:00.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:00 smithi184 ceph-mon[100622]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:00.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:00 smithi184 ceph-mon[102844]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:00.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:00.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:00.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:02 smithi184 ceph-mon[100622]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:02 smithi184 ceph-mon[102844]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:19:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:02 smithi196 ceph-mon[76349]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:03.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:03 smithi184 ceph-mon[100622]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:03.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:03 smithi184 ceph-mon[102844]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:03 smithi196 ceph-mon[76349]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:04.112 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:03.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:04.112 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:03.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:04.112 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:03.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:05 smithi184 ceph-mon[102844]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:05 smithi184 ceph-mon[100622]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:05 smithi196 ceph-mon[76349]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:07 smithi184 ceph-mon[100622]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:07 smithi184 ceph-mon[102844]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:07 smithi196 ceph-mon[76349]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:09.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:09 smithi184 ceph-mon[102844]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:09.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:09 smithi184 ceph-mon[100622]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:09 smithi196 ceph-mon[76349]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:10.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:10.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:10.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:10.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:10.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:11.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:11 smithi184 ceph-mon[100622]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:11.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:11 smithi184 ceph-mon[102844]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:11 smithi196 ceph-mon[76349]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:19:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:13 smithi184 ceph-mon[100622]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:13 smithi184 ceph-mon[102844]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:13 smithi196 ceph-mon[76349]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:14.158 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:13.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:14.158 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:13.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:14.158 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:13.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:15 smithi184 ceph-mon[100622]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:15 smithi184 ceph-mon[102844]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:15 smithi196 ceph-mon[76349]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:17 smithi196 ceph-mon[76349]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:17.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:17.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[102844]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[100622]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:19.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:19 smithi184 ceph-mon[100622]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:19.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:19 smithi184 ceph-mon[102844]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:19 smithi196 ceph-mon[76349]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:20.451 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:20.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:20.588 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:19:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:20.449Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:20.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:20.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:21.115 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 96m - - 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 95m - - 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (90m) 5s ago 91m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (91m) 3s ago 91m 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (84m) 5s ago 89m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (84m) 3s ago 95m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (84m) 5s ago 96m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (83m) 5s ago 97m 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:19:21.116 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (82m) 3s ago 95m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (83m) 5s ago 95m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (90m) 5s ago 90m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (90m) 3s ago 90m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (81m) 5s ago 94m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (80m) 5s ago 94m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (80m) 5s ago 94m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (79m) 5s ago 93m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (78m) 3s ago 93m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (77m) 3s ago 92m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (76m) 3s ago 92m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (75m) 3s ago 92m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:19:21.117 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (84m) 3s ago 91m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:19:21.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (75m) 5s ago 89m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:19:21.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (75m) 3s ago 89m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:19:21.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (75m) 5s ago 89m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:19:21.118 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (74m) 3s ago 89m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:19:21.427 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:19:21.427 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:19:21.427 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:19:21.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:19:21.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:19:21.429 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:19:21.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[100622]: from='client.49908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[100622]: from='client.49914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[100622]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:21.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[102844]: from='client.49908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[102844]: from='client.49914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:21 smithi184 ceph-mon[102844]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:21 smithi196 ceph-mon[76349]: from='client.49908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:21 smithi196 ceph-mon[76349]: from='client.49914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:21 smithi196 ceph-mon[76349]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:19:21.699 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:19:21.700 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:19:22.012 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:19:22.012 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:22.012 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[100622]: from='client.59885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1482873778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[100622]: from='client.59897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4212525204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[102844]: from='client.59885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1482873778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[102844]: from='client.59897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4212525204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:22 smithi196 ceph-mon[76349]: from='client.59885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1482873778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:22 smithi196 ceph-mon[76349]: from='client.59897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4212525204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:23.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:23 smithi184 ceph-mon[100622]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:23.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:23 smithi184 ceph-mon[102844]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:23 smithi196 ceph-mon[76349]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:23.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:23.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:23.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:25 smithi184 ceph-mon[100622]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:25 smithi184 ceph-mon[102844]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:25 smithi196 ceph-mon[76349]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:27 smithi184 ceph-mon[100622]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:27 smithi184 ceph-mon[102844]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:27 smithi196 ceph-mon[76349]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:29 smithi196 ceph-mon[76349]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:29 smithi184 ceph-mon[100622]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:29 smithi184 ceph-mon[102844]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:30.449Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:30.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:30.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:31 smithi184 ceph-mon[100622]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:31 smithi184 ceph-mon[102844]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:31 smithi196 ceph-mon[76349]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:19:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:33 smithi184 ceph-mon[100622]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:33 smithi184 ceph-mon[102844]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:33 smithi196 ceph-mon[76349]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:33.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:33.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:33.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:35 smithi184 ceph-mon[100622]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:35 smithi184 ceph-mon[102844]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:35 smithi196 ceph-mon[76349]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:37.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:37 smithi196 ceph-mon[76349]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:38.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[100622]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[102844]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:39 smithi196 ceph-mon[76349]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:39.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:39 smithi184 ceph-mon[100622]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:39.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:39 smithi184 ceph-mon[102844]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:40.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:40.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:40.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:40.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:40.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:41 smithi184 ceph-mon[100622]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:41 smithi184 ceph-mon[102844]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:41 smithi196 ceph-mon[76349]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:19:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:43 smithi184 ceph-mon[100622]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:43 smithi184 ceph-mon[102844]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:43 smithi196 ceph-mon[76349]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:44.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:43.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:44.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:43.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:44.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:43.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:44.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:44.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:45.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:45 smithi184 ceph-mon[100622]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:45.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:45 smithi184 ceph-mon[102844]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:45 smithi196 ceph-mon[76349]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:47 smithi184 ceph-mon[100622]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:47 smithi184 ceph-mon[102844]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:47 smithi196 ceph-mon[76349]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:19:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:49 smithi184 ceph-mon[100622]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:49 smithi184 ceph-mon[102844]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:49 smithi196 ceph-mon[76349]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:50.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:50.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:50.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:50.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:50.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:51 smithi184 ceph-mon[100622]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:51 smithi184 ceph-mon[102844]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:51 smithi196 ceph-mon[76349]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:19:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:19:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:19:52.844 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:19:52.844 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 96m - - 2024-09-20T21:19:52.844 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 96m - - 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (90m) 17s ago 91m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (91m) 15s ago 91m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (84m) 17s ago 89m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (84m) 15s ago 95m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (85m) 17s ago 97m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (83m) 17s ago 97m 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (83m) 15s ago 96m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (83m) 17s ago 96m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (91m) 17s ago 91m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (91m) 15s ago 91m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (82m) 17s ago 95m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (81m) 17s ago 95m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (80m) 17s ago 94m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:19:52.845 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (80m) 17s ago 94m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (78m) 15s ago 93m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (77m) 15s ago 93m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (76m) 15s ago 93m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (76m) 15s ago 92m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (85m) 15s ago 92m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (75m) 17s ago 90m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (75m) 15s ago 90m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (75m) 17s ago 89m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:19:52.846 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (75m) 15s ago 89m 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:19:53.154 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:19:53.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:19:53.156 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:19:53.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:19:53.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:19:53.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:19:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:19:53.426 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:19:53.426 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:19:53.426 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:19:53.426 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:19:53.426 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:19:53.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[102844]: from='client.59909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[102844]: from='client.59915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[102844]: from='client.49956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[102844]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:53.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2312867678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:53.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[100622]: from='client.59909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[100622]: from='client.59915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[100622]: from='client.49956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[100622]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:53.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2312867678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:53 smithi196 ceph-mon[76349]: from='client.59909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:53 smithi196 ceph-mon[76349]: from='client.59915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:53 smithi196 ceph-mon[76349]: from='client.49956 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:53 smithi196 ceph-mon[76349]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2312867678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:53.737 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:19:53.737 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:53.737 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:53.737 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:53.737 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:53.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:53.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:19:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:19:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:19:53.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:19:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:54 smithi184 ceph-mon[102844]: from='client.49968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/632061318' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:54 smithi184 ceph-mon[100622]: from='client.49968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/632061318' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:54 smithi196 ceph-mon[76349]: from='client.49968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/632061318' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:55 smithi184 ceph-mon[100622]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:55 smithi184 ceph-mon[102844]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:55 smithi196 ceph-mon[76349]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:57 smithi196 ceph-mon[76349]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[100622]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[102844]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:19:59.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:19:59 smithi196 ceph-mon[76349]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:19:59 smithi184 ceph-mon[102844]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:19:59 smithi184 ceph-mon[100622]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:00.452Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:00.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:00.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:01 smithi184 ceph-mon[100622]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:02.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:01 smithi184 ceph-mon[102844]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:02.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:20:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:01 smithi196 ceph-mon[76349]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:04.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:03.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:04.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:03.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:04.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:03.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:04 smithi184 ceph-mon[100622]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:04 smithi184 ceph-mon[102844]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:04 smithi196 ceph-mon[76349]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:06.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:06 smithi184 ceph-mon[100622]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:06.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:06 smithi184 ceph-mon[102844]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:06 smithi196 ceph-mon[76349]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:08 smithi184 ceph-mon[100622]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:08 smithi184 ceph-mon[102844]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:08 smithi196 ceph-mon[76349]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:10.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-mon[100622]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:10.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:10.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-mon[102844]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:10.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:10 smithi196 ceph-mon[76349]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:10.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:10.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:10.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:12 smithi184 ceph-mon[100622]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:12 smithi184 ceph-mon[102844]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:20:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:12 smithi196 ceph-mon[76349]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:14.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:13.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:14.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:13.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:14.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:13.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:14 smithi184 ceph-mon[102844]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:14 smithi184 ceph-mon[100622]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:14 smithi196 ceph-mon[76349]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[102844]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[100622]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:17.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:16 smithi196 ceph-mon[76349]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:18.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:18 smithi196 ceph-mon[76349]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:18.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:18.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:18.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[102844]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[100622]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-mon[100622]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-mon[102844]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:20.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:20.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:20.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:20 smithi196 ceph-mon[76349]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:20:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:22 smithi196 ceph-mon[76349]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:22 smithi184 ceph-mon[100622]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:22 smithi184 ceph-mon[102844]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:24.036 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:20:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:23.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:23.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:23.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 97m - - 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 96m - - 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (91m) 8s ago 92m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (92m) 6s ago 92m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (85m) 8s ago 90m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (85m) 6s ago 96m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:20:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (86m) 8s ago 97m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (84m) 8s ago 98m 195M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (83m) 6s ago 96m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (84m) 8s ago 96m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (91m) 8s ago 91m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (91m) 6s ago 91m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (82m) 8s ago 95m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (81m) 8s ago 95m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (81m) 8s ago 95m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (80m) 8s ago 94m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (79m) 6s ago 94m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (78m) 6s ago 94m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (77m) 6s ago 93m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (76m) 6s ago 93m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:20:24.563 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (85m) 6s ago 92m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:20:24.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (76m) 8s ago 90m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:20:24.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (76m) 6s ago 90m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:20:24.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (76m) 8s ago 90m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:20:24.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (76m) 6s ago 90m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:20:24.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:20:24.876 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:20:24.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:24 smithi184 ceph-mon[100622]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:24.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:24 smithi184 ceph-mon[100622]: from='client.59945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:24.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:24 smithi184 ceph-mon[102844]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:24.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:24 smithi184 ceph-mon[102844]: from='client.59945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:24 smithi196 ceph-mon[76349]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:24 smithi196 ceph-mon[76349]: from='client.59945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:20:25.146 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:20:25.147 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:20:25.462 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:25.462 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:25.462 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:25.462 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:25.462 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:25.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[100622]: from='client.49986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[100622]: from='client.59957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/869746167' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3219829959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[102844]: from='client.49986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[102844]: from='client.59957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/869746167' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:25.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3219829959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:25 smithi196 ceph-mon[76349]: from='client.49986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:25 smithi196 ceph-mon[76349]: from='client.59957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:25.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/869746167' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:25.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3219829959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:26.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:26 smithi184 ceph-mon[100622]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:26.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:26 smithi184 ceph-mon[100622]: from='client.59969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:26.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:26 smithi184 ceph-mon[102844]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:26.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:26 smithi184 ceph-mon[102844]: from='client.59969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:26 smithi196 ceph-mon[76349]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:26 smithi196 ceph-mon[76349]: from='client.59969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:28 smithi196 ceph-mon[76349]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:28 smithi184 ceph-mon[100622]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:28 smithi184 ceph-mon[102844]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:30.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:30.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-mon[100622]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-mon[102844]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:30.454Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:30.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:30.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:30.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:30 smithi196 ceph-mon[76349]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:30.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:20:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:32 smithi196 ceph-mon[76349]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:32 smithi184 ceph-mon[100622]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:32 smithi184 ceph-mon[102844]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:33.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:33.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:33.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:34 smithi196 ceph-mon[76349]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:34 smithi184 ceph-mon[100622]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:34 smithi184 ceph-mon[102844]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:36 smithi196 ceph-mon[76349]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[100622]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[102844]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:37.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:37.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:38 smithi196 ceph-mon[76349]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:39.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[100622]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[102844]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:40.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:40.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:40.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:40.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:40.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:40.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:40.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:40 smithi196 ceph-mon[76349]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:40.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-mon[100622]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-mon[102844]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:20:43.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:42 smithi184 ceph-mon[100622]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:43.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:42 smithi184 ceph-mon[102844]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:42 smithi196 ceph-mon[76349]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:43.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:43.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:43.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:44 smithi184 ceph-mon[100622]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:44 smithi184 ceph-mon[102844]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:44 smithi196 ceph-mon[76349]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:46 smithi184 ceph-mon[100622]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:46 smithi184 ceph-mon[102844]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:46 smithi196 ceph-mon[76349]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:48.996 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:48 smithi184 ceph-mon[100622]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:48.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:48 smithi184 ceph-mon[102844]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:48 smithi196 ceph-mon[76349]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:20:50.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:50.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:50.725 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:50.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:50.725 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:50.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-mon[100622]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-mon[102844]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:50 smithi196 ceph-mon[76349]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:20:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:20:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:52 smithi184 ceph-mon[100622]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:52 smithi184 ceph-mon[102844]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:52 smithi196 ceph-mon[76349]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:53.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:53.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:20:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:20:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:20:53.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:20:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:54 smithi184 ceph-mon[100622]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:54 smithi184 ceph-mon[102844]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:54 smithi196 ceph-mon[76349]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:55.761 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 97m - - 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 97m - - 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (91m) 0s ago 92m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (92m) 18s ago 92m 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (85m) 0s ago 90m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:20:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (85m) 18s ago 96m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (86m) 0s ago 98m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (84m) 0s ago 98m 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (84m) 18s ago 97m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (84m) 0s ago 97m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (92m) 0s ago 92m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (92m) 18s ago 92m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (83m) 0s ago 96m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (82m) 0s ago 96m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (81m) 0s ago 95m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (81m) 0s ago 95m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (79m) 18s ago 94m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (79m) 18s ago 94m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (77m) 18s ago 94m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:20:56.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (77m) 18s ago 93m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:20:56.298 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (86m) 18s ago 93m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:20:56.298 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (76m) 0s ago 91m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:20:56.298 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (76m) 18s ago 91m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:20:56.298 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (76m) 0s ago 91m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:20:56.298 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (76m) 18s ago 91m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:20:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:20:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:20:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:20:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:20:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:20:56.607 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:20:56.877 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:20:56.878 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:20:56.878 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:56.878 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='client.50016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:56.878 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='client.59987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2094970276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='client.50016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='client.59987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:56.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2094970276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='client.50016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='client.59987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2094970276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:57.192 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:57.192 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:57.192 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[100622]: from='client.50028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[100622]: from='client.60005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3669462311' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[102844]: from='client.50028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[102844]: from='client.60005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3669462311' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:57 smithi196 ceph-mon[76349]: from='client.50028 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:57 smithi196 ceph-mon[76349]: from='client.60005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3669462311' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:59.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:58 smithi196 ceph-mon[76349]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:59.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:20:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[100622]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[102844]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:20:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:20:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:00.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:00.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:00.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:00 smithi196 ceph-mon[76349]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-mon[100622]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-mon[102844]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:21:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:02 smithi196 ceph-mon[76349]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:02 smithi184 ceph-mon[100622]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:02 smithi184 ceph-mon[102844]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:03.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:03.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:03.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:04 smithi196 ceph-mon[76349]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:04 smithi184 ceph-mon[100622]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:04 smithi184 ceph-mon[102844]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:06 smithi196 ceph-mon[76349]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:07.180 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:06 smithi184 ceph-mon[100622]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:07.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:06 smithi184 ceph-mon[102844]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:08 smithi196 ceph-mon[76349]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:09.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:08 smithi184 ceph-mon[100622]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:09.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:08 smithi184 ceph-mon[102844]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:10.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:10.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:10.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:10 smithi196 ceph-mon[76349]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-mon[100622]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-mon[102844]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:21:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:12 smithi196 ceph-mon[76349]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:12 smithi184 ceph-mon[100622]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:12 smithi184 ceph-mon[102844]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:13.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:13.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:13.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:14 smithi196 ceph-mon[76349]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:14 smithi184 ceph-mon[100622]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:14 smithi184 ceph-mon[102844]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:16 smithi196 ceph-mon[76349]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[100622]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[102844]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[100622]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[102844]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:18 smithi196 ceph-mon[76349]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:20.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:20.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:20.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-mon[100622]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-mon[102844]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:21.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:20 smithi196 ceph-mon[76349]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:21.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:21:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:22 smithi196 ceph-mon[76349]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:22 smithi184 ceph-mon[100622]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:22 smithi184 ceph-mon[102844]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:23.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:23.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:23.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:24 smithi184 ceph-mon[100622]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:24 smithi184 ceph-mon[102844]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:24 smithi196 ceph-mon[76349]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:26 smithi184 ceph-mon[100622]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:26 smithi184 ceph-mon[102844]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:26 smithi196 ceph-mon[76349]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:27.488 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 98m - - 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 97m - - 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (92m) 11s ago 93m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (93m) 10s ago 93m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (86m) 11s ago 91m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (86m) 10s ago 97m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (87m) 11s ago 99m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (85m) 11s ago 99m 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (84m) 10s ago 97m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:21:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (85m) 11s ago 97m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (93m) 11s ago 93m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (92m) 10s ago 92m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (83m) 11s ago 97m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (82m) 11s ago 96m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (82m) 11s ago 96m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (81m) 11s ago 95m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (80m) 10s ago 95m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (79m) 10s ago 95m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (78m) 10s ago 94m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (77m) 10s ago 94m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (86m) 10s ago 93m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (77m) 11s ago 91m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (77m) 10s ago 92m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:21:28.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (77m) 11s ago 91m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:21:28.042 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (77m) 10s ago 91m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:21:28.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:21:28.356 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:21:28.625 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:21:28.626 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:21:28.949 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:21:28.949 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:28.950 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:29.242 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='client.60017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='client.60023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='client.50064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3091045988' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/393371037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='client.60017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='client.60023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='client.50064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3091045988' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:29.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:29.244 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/393371037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='client.60017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='client.60023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='client.50064 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3091045988' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/393371037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:30 smithi184 ceph-mon[100622]: from='client.60041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:30 smithi184 ceph-mon[102844]: from='client.60041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:30 smithi196 ceph-mon[76349]: from='client.60041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:30.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:30.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:30.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:31 smithi184 ceph-mon[102844]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:31 smithi184 ceph-mon[100622]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:31 smithi196 ceph-mon[76349]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:21:33.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:33 smithi184 ceph-mon[100622]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:33.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:33 smithi184 ceph-mon[102844]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:33 smithi196 ceph-mon[76349]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:33.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:33.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:33.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:35 smithi184 ceph-mon[100622]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:35 smithi184 ceph-mon[102844]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:35 smithi196 ceph-mon[76349]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[100622]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[102844]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:37 smithi196 ceph-mon[76349]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:37.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:39 smithi196 ceph-mon[76349]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[100622]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[102844]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:40.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:40.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:40.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:41 smithi196 ceph-mon[76349]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:41 smithi184 ceph-mon[100622]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:41 smithi184 ceph-mon[102844]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:42.260 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:21:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:43 smithi196 ceph-mon[76349]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:43 smithi184 ceph-mon[100622]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:43 smithi184 ceph-mon[102844]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:43.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:43.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:43.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:45 smithi196 ceph-mon[76349]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:45 smithi184 ceph-mon[100622]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:45 smithi184 ceph-mon[102844]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:47 smithi196 ceph-mon[76349]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:47 smithi184 ceph-mon[100622]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:47 smithi184 ceph-mon[102844]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:48.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:48 smithi184 ceph-mon[100622]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:48.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:48 smithi184 ceph-mon[102844]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:48 smithi196 ceph-mon[76349]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:50.386 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-mon[100622]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:50.386 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:50.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-mon[102844]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:50.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:50 smithi196 ceph-mon[76349]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:50.691 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:50.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:50.692 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:50.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:50.692 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:50.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:52 smithi184 ceph-mon[100622]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:52 smithi184 ceph-mon[102844]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:21:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:21:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:52 smithi196 ceph-mon[76349]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:54.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:53.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:54.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:53.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:21:54.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:21:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:21:53.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:21:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:54 smithi184 ceph-mon[100622]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:54 smithi184 ceph-mon[102844]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:54 smithi196 ceph-mon[76349]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:56 smithi184 ceph-mon[100622]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:56 smithi184 ceph-mon[102844]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:56 smithi196 ceph-mon[76349]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:57.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:58 smithi196 ceph-mon[76349]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[100622]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[102844]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:21:59.252 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 98m - - 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 98m - - 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (92m) 3s ago 93m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (93m) 1s ago 93m 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (86m) 3s ago 91m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (86m) 1s ago 97m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:21:59.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (87m) 3s ago 99m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (85m) 3s ago 99m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (85m) 1s ago 98m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (85m) 3s ago 98m 143M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (93m) 3s ago 93m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (93m) 1s ago 93m 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (84m) 3s ago 97m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (83m) 3s ago 97m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (82m) 3s ago 96m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (82m) 3s ago 96m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (80m) 1s ago 95m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (80m) 1s ago 95m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (78m) 1s ago 95m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (78m) 1s ago 94m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:21:59.785 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (87m) 1s ago 94m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:21:59.786 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (77m) 3s ago 92m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:21:59.786 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (77m) 1s ago 92m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:21:59.786 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (77m) 3s ago 92m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:21:59.786 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (77m) 1s ago 92m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:22:00.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:00.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:22:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:22:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:22:00.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:22:00.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:22:00.366 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:22:00.367 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:22:00.643 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:00.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:00.643 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:00.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:00.643 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:00.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:00.681 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:22:00.681 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:00.681 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:00.681 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:00.681 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: from='client.60053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: from='client.50094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: from='client.60065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:00.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1012558497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: from='client.60053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: from='client.50094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: from='client.60065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1012558497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: from='client.60053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: from='client.50094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: from='client.60065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1012558497' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:01 smithi184 ceph-mon[100622]: from='client.60077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1193641784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:01 smithi184 ceph-mon[102844]: from='client.60077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1193641784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:01 smithi196 ceph-mon[76349]: from='client.60077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:02.163 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1193641784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:22:02.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:02 smithi196 ceph-mon[76349]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:02 smithi184 ceph-mon[100622]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:02 smithi184 ceph-mon[102844]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:03.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:03.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:03.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:04 smithi184 ceph-mon[100622]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:04 smithi184 ceph-mon[102844]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:04 smithi196 ceph-mon[76349]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:06 smithi184 ceph-mon[100622]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:06 smithi184 ceph-mon[102844]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:06 smithi196 ceph-mon[76349]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:08 smithi184 ceph-mon[100622]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:08 smithi184 ceph-mon[102844]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:08 smithi196 ceph-mon[76349]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-mon[100622]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-mon[102844]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:10.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:10.461Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:10.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:10.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:10 smithi196 ceph-mon[76349]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:22:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:12 smithi184 ceph-mon[100622]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:12 smithi184 ceph-mon[102844]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:12 smithi196 ceph-mon[76349]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:13.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:13.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:13.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:14 smithi184 ceph-mon[100622]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:14 smithi184 ceph-mon[102844]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:14 smithi196 ceph-mon[76349]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:16 smithi184 ceph-mon[100622]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:16 smithi184 ceph-mon[102844]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:16 smithi196 ceph-mon[76349]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[102844]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[100622]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:18 smithi196 ceph-mon[76349]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:22:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:20.771 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:20.463Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:20.771 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:20.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:20.772 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:20.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-mon[100622]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-mon[102844]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:20 smithi196 ceph-mon[76349]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:22:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:22 smithi184 ceph-mon[100622]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:22 smithi184 ceph-mon[102844]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:23.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:22 smithi196 ceph-mon[76349]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:23.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:23.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:23.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:24 smithi184 ceph-mon[100622]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:24 smithi184 ceph-mon[102844]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:24 smithi196 ceph-mon[76349]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:26 smithi184 ceph-mon[100622]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:26 smithi184 ceph-mon[102844]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:26 smithi196 ceph-mon[76349]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:28 smithi184 ceph-mon[100622]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:28 smithi184 ceph-mon[102844]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:28 smithi196 ceph-mon[76349]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:30.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:30.463Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:30.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:30.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:30.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:30.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:30.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-mon[100622]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:30.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:30.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-mon[102844]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:30.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:30.997 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:22:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:30 smithi196 ceph-mon[76349]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 99m - - 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 99m - - 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (93m) 14s ago 94m 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (94m) 13s ago 94m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (87m) 14s ago 92m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (87m) 13s ago 98m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (88m) 14s ago 100m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (86m) 14s ago 100m 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:22:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (85m) 13s ago 98m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (86m) 14s ago 98m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (94m) 14s ago 94m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (94m) 13s ago 94m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (84m) 14s ago 98m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (83m) 14s ago 97m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (83m) 14s ago 97m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (82m) 14s ago 96m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (81m) 13s ago 96m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (80m) 13s ago 96m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (79m) 13s ago 95m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (78m) 13s ago 95m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (87m) 13s ago 94m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (78m) 14s ago 93m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (78m) 13s ago 93m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:22:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (78m) 14s ago 92m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:22:31.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (78m) 13s ago 92m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:22:31.875 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:22:31.876 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:22:31.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:22:31.877 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:22:31.877 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:22:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:31 smithi184 ceph-mon[100622]: from='client.60089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:31 smithi184 ceph-mon[102844]: from='client.60089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:22:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:31 smithi196 ceph-mon[76349]: from='client.60089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:32.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:22:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:22:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:22:32.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:22:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:22:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:32.465 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:32.465 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[100622]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[100622]: from='client.50130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[100622]: from='client.50136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3524578681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2702681026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[102844]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[102844]: from='client.50130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[102844]: from='client.50136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3524578681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2702681026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:33.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:32 smithi196 ceph-mon[76349]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:32 smithi196 ceph-mon[76349]: from='client.50130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:32 smithi196 ceph-mon[76349]: from='client.50136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:33.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3524578681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:33.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2702681026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:33 smithi196 ceph-mon[76349]: from='client.60113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:34.246 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:33 smithi184 ceph-mon[102844]: from='client.60113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:34.247 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:33 smithi184 ceph-mon[100622]: from='client.60113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:34.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:33.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:34.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:33.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:34.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:33.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:34 smithi196 ceph-mon[76349]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:34 smithi184 ceph-mon[100622]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:34 smithi184 ceph-mon[102844]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:36 smithi196 ceph-mon[76349]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:36 smithi184 ceph-mon[100622]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:36 smithi184 ceph-mon[102844]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[102844]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[100622]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:38 smithi196 ceph-mon[76349]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:40.464Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:40.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:40.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-mon[100622]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-mon[102844]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:40 smithi196 ceph-mon[76349]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:22:43.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:42 smithi196 ceph-mon[76349]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:42 smithi184 ceph-mon[100622]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:42 smithi184 ceph-mon[102844]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:43.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:43.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:44.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:43.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:44.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:44 smithi196 ceph-mon[76349]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:44 smithi184 ceph-mon[100622]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:44 smithi184 ceph-mon[102844]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:46 smithi184 ceph-mon[100622]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:46 smithi184 ceph-mon[102844]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:46 smithi196 ceph-mon[76349]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:48 smithi184 ceph-mon[100622]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:48 smithi184 ceph-mon[102844]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:48 smithi196 ceph-mon[76349]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:50.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:50.464Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:50.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:50.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:50.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:50.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-mon[100622]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-mon[102844]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:50 smithi196 ceph-mon[76349]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:22:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:22:53.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:52 smithi184 ceph-mon[100622]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:53.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:52 smithi184 ceph-mon[102844]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:52 smithi196 ceph-mon[76349]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:54.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:53.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:54.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:53.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:22:54.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:22:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:22:53.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:22:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:54 smithi196 ceph-mon[76349]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:54 smithi184 ceph-mon[100622]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:54 smithi184 ceph-mon[102844]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:56 smithi184 ceph-mon[100622]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:56 smithi184 ceph-mon[102844]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:56 smithi196 ceph-mon[76349]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[100622]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[102844]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:59 smithi196 ceph-mon[76349]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:22:59.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:00.465Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:00.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:00.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:01 smithi184 ceph-mon[100622]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:01 smithi184 ceph-mon[102844]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:01 smithi196 ceph-mon[76349]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:23:02.761 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:23:03.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:03 smithi196 ceph-mon[76349]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:03 smithi184 ceph-mon[100622]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:03 smithi184 ceph-mon[102844]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 99m - - 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 99m - - 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (93m) 6s ago 95m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (94m) 4s ago 94m 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (87m) 6s ago 93m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:23:03.307 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (87m) 4s ago 98m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (88m) 6s ago 100m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (87m) 6s ago 100m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (86m) 4s ago 99m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (86m) 6s ago 99m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (94m) 6s ago 94m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (94m) 4s ago 94m 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (85m) 6s ago 98m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (84m) 6s ago 98m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (83m) 6s ago 97m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (83m) 6s ago 97m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (81m) 4s ago 97m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (81m) 4s ago 96m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:23:03.308 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (80m) 4s ago 96m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (79m) 4s ago 95m 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (88m) 4s ago 95m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (79m) 6s ago 93m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (78m) 4s ago 93m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (78m) 6s ago 93m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:23:03.309 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (78m) 4s ago 93m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:23:03.627 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:23:03.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:23:03.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:23:03.629 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:23:03.912 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:23:03.913 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:23:03.913 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:23:03.913 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:23:04.224 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:23:04.224 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:04.225 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:04.225 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:04.225 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:04.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[100622]: from='client.60125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[100622]: from='client.60131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3494990951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:04.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[102844]: from='client.60125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[102844]: from='client.60131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3494990951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:04.226 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:03.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:04.226 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:03.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:04.226 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:03.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:04.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:04 smithi196 ceph-mon[76349]: from='client.60125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:04 smithi196 ceph-mon[76349]: from='client.60131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:04.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3494990951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:05 smithi196 ceph-mon[76349]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:05 smithi196 ceph-mon[76349]: from='client.67999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:05 smithi196 ceph-mon[76349]: from='client.60149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1926066407' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[100622]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[100622]: from='client.67999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[100622]: from='client.60149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1926066407' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[102844]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[102844]: from='client.67999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[102844]: from='client.60149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1926066407' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:07 smithi196 ceph-mon[76349]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:07 smithi184 ceph-mon[100622]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:07 smithi184 ceph-mon[102844]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:09 smithi196 ceph-mon[76349]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:09 smithi184 ceph-mon[100622]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:09 smithi184 ceph-mon[102844]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:10.466Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:10.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:10.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:11.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:11 smithi184 ceph-mon[100622]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:11.332 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:11.332 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:11 smithi184 ceph-mon[102844]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:11.332 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:11 smithi196 ceph-mon[76349]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:12.230 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:23:13.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:13 smithi184 ceph-mon[100622]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:13.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:13 smithi184 ceph-mon[102844]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:13 smithi196 ceph-mon[76349]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:13.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:13.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:14.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:13.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:14.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:14.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:15 smithi196 ceph-mon[76349]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:15 smithi184 ceph-mon[100622]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:15 smithi184 ceph-mon[102844]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:17 smithi196 ceph-mon[76349]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:17 smithi184 ceph-mon[100622]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:17 smithi184 ceph-mon[102844]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[100622]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:18 smithi184 ceph-mon[102844]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:18 smithi196 ceph-mon[76349]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:20.771 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:20.467Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:20.772 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:20.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:20.772 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:20.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-mon[100622]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-mon[102844]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:20 smithi196 ceph-mon[76349]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:23:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:22 smithi184 ceph-mon[100622]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:22 smithi184 ceph-mon[102844]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:22 smithi196 ceph-mon[76349]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:23.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:23.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:23.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:24.986 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:24 smithi196 ceph-mon[76349]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:24 smithi184 ceph-mon[100622]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:24 smithi184 ceph-mon[102844]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:26 smithi196 ceph-mon[76349]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:26 smithi184 ceph-mon[102844]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:26 smithi184 ceph-mon[100622]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:28 smithi184 ceph-mon[100622]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:28 smithi184 ceph-mon[102844]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:28 smithi196 ceph-mon[76349]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:30.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:30.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:30.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:30.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:30.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:30.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-mon[100622]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-mon[102844]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:30 smithi196 ceph-mon[76349]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:32.097 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:23:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:32 smithi184 ceph-mon[100622]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:32 smithi184 ceph-mon[102844]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:32 smithi196 ceph-mon[76349]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:33.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:33.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:33.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:34.520 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:23:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:34 smithi184 ceph-mon[100622]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:34 smithi184 ceph-mon[102844]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:35.081 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 100m - - 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 100m - - 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (94m) 17s ago 95m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (95m) 16s ago 95m 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (88m) 17s ago 93m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (88m) 16s ago 99m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (89m) 17s ago 101m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (87m) 17s ago 101m 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (86m) 16s ago 99m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (87m) 17s ago 99m 143M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (95m) 17s ago 95m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:23:35.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (95m) 16s ago 95m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (85m) 17s ago 99m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (84m) 17s ago 98m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (84m) 17s ago 98m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (83m) 17s ago 97m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (82m) 16s ago 97m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (81m) 16s ago 97m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (80m) 16s ago 96m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (79m) 16s ago 96m 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (88m) 16s ago 95m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (79m) 17s ago 94m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (79m) 16s ago 94m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (79m) 17s ago 93m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:23:35.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (79m) 16s ago 93m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:23:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:34 smithi196 ceph-mon[76349]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:23:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:23:35.401 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:23:35.667 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:23:35.668 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:23:35.981 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:23:35.981 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:35.981 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:35.981 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:35.981 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[100622]: from='client.60161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[100622]: from='client.60167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[100622]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1718608997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[102844]: from='client.60161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[102844]: from='client.60167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[102844]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:35.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1718608997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:35 smithi196 ceph-mon[76349]: from='client.60161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:35 smithi196 ceph-mon[76349]: from='client.60167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:35 smithi196 ceph-mon[76349]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1718608997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:36 smithi196 ceph-mon[76349]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:36 smithi196 ceph-mon[76349]: from='client.50217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3495185181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[100622]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[100622]: from='client.50217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3495185181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[102844]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[102844]: from='client.50217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:37.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3495185181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[100622]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[102844]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:38 smithi196 ceph-mon[76349]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:40.469Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:40.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:40.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-mon[100622]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-mon[102844]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:40 smithi196 ceph-mon[76349]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:23:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:42 smithi184 ceph-mon[100622]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:42 smithi184 ceph-mon[102844]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:42 smithi196 ceph-mon[76349]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:43.905Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:43.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:43.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:44.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:44 smithi196 ceph-mon[76349]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:44 smithi184 ceph-mon[100622]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:44 smithi184 ceph-mon[102844]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:46 smithi196 ceph-mon[76349]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:46 smithi184 ceph-mon[100622]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:46 smithi184 ceph-mon[102844]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:48 smithi184 ceph-mon[100622]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:48 smithi184 ceph-mon[102844]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:48 smithi196 ceph-mon[76349]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:50.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:50.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:50.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-mon[100622]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-mon[102844]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:50 smithi196 ceph-mon[76349]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:23:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:23:53.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:52 smithi184 ceph-mon[100622]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:53.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:52 smithi184 ceph-mon[102844]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:53.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:52 smithi196 ceph-mon[76349]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:53.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:53.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:23:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:23:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:23:53.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:23:55.105 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:54 smithi184 ceph-mon[100622]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:55.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:54 smithi184 ceph-mon[102844]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:54 smithi196 ceph-mon[76349]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:56 smithi184 ceph-mon[100622]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:56 smithi184 ceph-mon[102844]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:56 smithi196 ceph-mon[76349]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[100622]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[102844]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:58 smithi196 ceph-mon[76349]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:59.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:59.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:23:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:00.471Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:00.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:00.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:01 smithi184 ceph-mon[100622]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:01 smithi184 ceph-mon[102844]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:01 smithi196 ceph-mon[76349]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:24:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:03 smithi184 ceph-mon[100622]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:03 smithi184 ceph-mon[102844]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:03 smithi196 ceph-mon[76349]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:03.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:03.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:03.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:05 smithi184 ceph-mon[100622]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:05 smithi184 ceph-mon[102844]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:05.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:05 smithi196 ceph-mon[76349]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:06.267 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 100m - - 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 100m - - 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (95m) 9s ago 96m 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (95m) 7s ago 95m 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (89m) 9s ago 94m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (88m) 7s ago 99m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (89m) 9s ago 101m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (88m) 9s ago 101m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:24:06.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (87m) 7s ago 100m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (87m) 9s ago 100m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (95m) 9s ago 95m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (95m) 7s ago 95m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (86m) 9s ago 99m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (85m) 9s ago 99m 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (84m) 9s ago 98m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (84m) 9s ago 98m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (83m) 7s ago 98m 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (82m) 7s ago 97m 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (81m) 7s ago 97m 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (80m) 7s ago 96m 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (89m) 7s ago 96m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (80m) 9s ago 94m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:24:06.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (79m) 7s ago 94m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:24:06.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (79m) 9s ago 94m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:24:06.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (79m) 7s ago 94m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:24:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:07 smithi184 ceph-mon[100622]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:24:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:07 smithi184 ceph-mon[102844]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:24:07.108 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:24:07.108 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:24:07.108 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:24:07.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:24:07.109 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:24:07.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:07 smithi196 ceph-mon[76349]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:24:07.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:24:07.380 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:24:07.380 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:24:07.380 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:24:07.381 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:24:07.690 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:24:07.690 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:07.690 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:07.690 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:07.690 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:08 smithi196 ceph-mon[76349]: from='client.60197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:08 smithi196 ceph-mon[76349]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:08 smithi196 ceph-mon[76349]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2753528949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2355241999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[100622]: from='client.60197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[100622]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[100622]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2753528949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2355241999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[102844]: from='client.60197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[102844]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[102844]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2753528949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2355241999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:09 smithi184 ceph-mon[100622]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:24:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:09 smithi184 ceph-mon[100622]: from='client.60221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:09 smithi184 ceph-mon[102844]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:24:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:09 smithi184 ceph-mon[102844]: from='client.60221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:09 smithi196 ceph-mon[76349]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:24:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:09 smithi196 ceph-mon[76349]: from='client.60221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:10.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:10.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:10.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:11 smithi184 ceph-mon[100622]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:24:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:11 smithi184 ceph-mon[102844]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:11 smithi196 ceph-mon[76349]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:24:13.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:13 smithi184 ceph-mon[100622]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:13.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:13 smithi184 ceph-mon[102844]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:13 smithi196 ceph-mon[76349]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:14.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:14.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:14.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:13.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:14.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:13.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:14.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:13.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:15.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:15 smithi184 ceph-mon[100622]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:15.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:15 smithi184 ceph-mon[102844]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:15 smithi196 ceph-mon[76349]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:17 smithi196 ceph-mon[76349]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:24:17.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:17 smithi184 ceph-mon[100622]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:24:17.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:17 smithi184 ceph-mon[102844]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T21:24:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[102844]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[100622]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:18 smithi196 ceph-mon[76349]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[100622]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[102844]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:24:20.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:20 smithi196 ceph-mon[76349]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:24:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:20.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:20.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:20.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:22 smithi184 ceph-mon[100622]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:24:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T21:24:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:22 smithi184 ceph-mon[102844]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:24:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:22 smithi196 ceph-mon[76349]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:24:24.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:23.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:24.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:23.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:24.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:23.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:24 smithi184 ceph-mon[100622]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:24 smithi184 ceph-mon[102844]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:24 smithi196 ceph-mon[76349]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:24:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:26 smithi184 ceph-mon[100622]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:24:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:26 smithi184 ceph-mon[102844]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:24:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:26 smithi196 ceph-mon[76349]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:24:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:28 smithi196 ceph-mon[76349]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:24:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:28 smithi184 ceph-mon[100622]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:24:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:28 smithi184 ceph-mon[102844]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T21:24:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-mon[100622]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:24:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-mon[102844]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:24:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:30.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:30.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:30.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:30.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:30.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:30 smithi196 ceph-mon[76349]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:24:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:32.271 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:24:32.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:32 smithi184 ceph-mon[100622]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:24:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:32 smithi184 ceph-mon[102844]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:24:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:32 smithi196 ceph-mon[76349]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:24:34.120 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:33.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:34.120 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:33.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:34.120 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:33.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:34 smithi184 ceph-mon[100622]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T21:24:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:34 smithi184 ceph-mon[102844]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T21:24:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:34 smithi196 ceph-mon[76349]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-20T21:24:36.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:36 smithi184 ceph-mon[100622]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:24:36.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:36 smithi184 ceph-mon[102844]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:24:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:36 smithi196 ceph-mon[76349]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:24:37.975 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:24:38.500 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 101m - - 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 101m - - 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (95m) 0s ago 96m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (96m) 19s ago 96m 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (89m) 0s ago 94m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (89m) 19s ago 100m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (90m) 0s ago 102m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (88m) 0s ago 102m 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (88m) 19s ago 100m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (88m) 0s ago 100m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (96m) 0s ago 96m 16.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (96m) 19s ago 96m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:24:38.501 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (86m) 0s ago 100m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (86m) 0s ago 99m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (85m) 0s ago 99m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (84m) 0s ago 99m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (83m) 19s ago 98m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (82m) 19s ago 98m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (81m) 19s ago 97m 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (80m) 19s ago 97m 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (89m) 19s ago 97m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (80m) 0s ago 95m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (80m) 19s ago 95m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (80m) 0s ago 94m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:24:38.502 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (80m) 19s ago 94m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[100622]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[100622]: from='client.50265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[102844]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:38 smithi184 ceph-mon[102844]: from='client.50265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:24:38.817 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:24:39.084 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:24:39.085 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:24:39.085 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:24:39.085 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:24:39.085 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:24:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:38 smithi196 ceph-mon[76349]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:24:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:38 smithi196 ceph-mon[76349]: from='client.50265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:39.405 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:24:39.405 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:39.405 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:39.405 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:39.405 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='client.60239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='client.50277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/590567692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='client.60257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1542072215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='client.60239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='client.50277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/590567692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='client.60257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1542072215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='client.60239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='client.50277 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/590567692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='client.60257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:40.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1542072215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-mon[102844]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:24:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-mon[100622]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:24:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:40.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:40.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:40.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:40 smithi196 ceph-mon[76349]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:24:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:24:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:42 smithi184 ceph-mon[100622]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:24:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:42 smithi184 ceph-mon[102844]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:24:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:42 smithi196 ceph-mon[76349]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T21:24:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:43.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:43.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:43.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:44 smithi184 ceph-mon[100622]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:44 smithi184 ceph-mon[102844]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:45.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:45.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:44 smithi196 ceph-mon[76349]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:45.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:46 smithi196 ceph-mon[76349]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:46 smithi184 ceph-mon[100622]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:46 smithi184 ceph-mon[102844]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:48.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:48 smithi196 ceph-mon[76349]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:48 smithi184 ceph-mon[100622]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:48 smithi184 ceph-mon[102844]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-mon[100622]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-mon[102844]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:50.474Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:50.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:50.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:50 smithi196 ceph-mon[76349]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:24:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:24:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:52 smithi184 ceph-mon[100622]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:52 smithi184 ceph-mon[102844]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:52 smithi196 ceph-mon[76349]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:53.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:53.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:24:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:24:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:24:53.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:24:54.915 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:54 smithi184 ceph-mon[100622]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:54.916 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:54 smithi184 ceph-mon[102844]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:54 smithi196 ceph-mon[76349]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:56 smithi184 ceph-mon[100622]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:56 smithi184 ceph-mon[102844]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:56 smithi196 ceph-mon[76349]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[100622]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[102844]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:58 smithi196 ceph-mon[76349]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[102844]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[100622]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:00.475Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:00.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:00.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:00 smithi196 ceph-mon[76349]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:25:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:02 smithi184 ceph-mon[100622]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:02 smithi184 ceph-mon[102844]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:02 smithi196 ceph-mon[76349]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:03.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:03.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:03.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:04 smithi184 ceph-mon[100622]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:04 smithi184 ceph-mon[102844]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:04 smithi196 ceph-mon[76349]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:06 smithi184 ceph-mon[100622]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:06 smithi184 ceph-mon[102844]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:06 smithi196 ceph-mon[76349]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:08 smithi196 ceph-mon[76349]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:08 smithi184 ceph-mon[100622]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:08 smithi184 ceph-mon[102844]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 101m - - 2024-09-20T21:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 101m - - 2024-09-20T21:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (96m) 12s ago 97m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (96m) 10s ago 96m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (90m) 12s ago 95m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (89m) 10s ago 101m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (90m) 12s ago 102m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (89m) 12s ago 102m 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (88m) 10s ago 101m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (88m) 12s ago 101m 143M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (96m) 12s ago 96m 16.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (96m) 10s ago 96m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (87m) 12s ago 100m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (86m) 12s ago 100m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (85m) 12s ago 99m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (85m) 12s ago 99m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (84m) 10s ago 99m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (83m) 10s ago 98m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (82m) 10s ago 98m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (81m) 10s ago 98m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (90m) 10s ago 97m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (81m) 12s ago 95m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (80m) 10s ago 95m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (80m) 12s ago 95m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (80m) 10s ago 95m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:25:10.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:25:10.534 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:25:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[102844]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[102844]: from='client.60269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[102844]: from='client.60275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1134435987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[100622]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[100622]: from='client.60269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[100622]: from='client.60275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1134435987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:10.476Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:10.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:10.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:25:10.801 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:25:10.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:25:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:10 smithi196 ceph-mon[76349]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:10 smithi196 ceph-mon[76349]: from='client.60269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:10 smithi196 ceph-mon[76349]: from='client.60275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1134435987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:11.112 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:25:11.112 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:11.112 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:11.112 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:11.112 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:11 smithi196 ceph-mon[76349]: from='client.60281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:11 smithi196 ceph-mon[76349]: from='client.50325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:11.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4018247798' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[100622]: from='client.60281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[100622]: from='client.50325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4018247798' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[102844]: from='client.60281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[102844]: from='client.50325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4018247798' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:25:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:12 smithi196 ceph-mon[76349]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:12 smithi184 ceph-mon[100622]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:12 smithi184 ceph-mon[102844]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:13.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:13.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:13.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:14 smithi196 ceph-mon[76349]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:14 smithi184 ceph-mon[100622]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:14 smithi184 ceph-mon[102844]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:16.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:16 smithi184 ceph-mon[100622]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:16.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:16 smithi184 ceph-mon[102844]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:16 smithi196 ceph-mon[76349]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[100622]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[102844]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:18 smithi196 ceph-mon[76349]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[102844]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[100622]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:20.477Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:20.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:20.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:20 smithi196 ceph-mon[76349]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:25:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:22 smithi196 ceph-mon[76349]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:22 smithi184 ceph-mon[100622]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:22 smithi184 ceph-mon[102844]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:23.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:23.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:23.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:24 smithi196 ceph-mon[76349]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:24 smithi184 ceph-mon[100622]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:24 smithi184 ceph-mon[102844]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:26 smithi196 ceph-mon[76349]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:26 smithi184 ceph-mon[100622]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:26 smithi184 ceph-mon[102844]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:28.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:28 smithi196 ceph-mon[76349]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:28 smithi184 ceph-mon[100622]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:28 smithi184 ceph-mon[102844]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:29.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:29.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-mon[100622]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-mon[102844]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:30.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:30.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:30.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:30 smithi196 ceph-mon[76349]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:25:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:32 smithi196 ceph-mon[76349]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:32 smithi184 ceph-mon[102844]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:32 smithi184 ceph-mon[100622]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:33.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:33.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:33.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:34 smithi196 ceph-mon[76349]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:34 smithi184 ceph-mon[100622]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:34 smithi184 ceph-mon[102844]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:36 smithi184 ceph-mon[100622]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:36 smithi184 ceph-mon[102844]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:36 smithi196 ceph-mon[76349]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[100622]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[102844]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:39 smithi196 ceph-mon[76349]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[100622]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[102844]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:40.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:40.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:40.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:40 smithi196 ceph-mon[76349]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:41.410 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:25:41.945 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:25:41.945 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 102m - - 2024-09-20T21:25:41.945 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 102m - - 2024-09-20T21:25:41.945 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (96m) 3s ago 97m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:25:41.945 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (97m) 2s ago 97m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (90m) 3s ago 95m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (90m) 2s ago 101m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (91m) 3s ago 103m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (89m) 3s ago 103m 202M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (89m) 2s ago 101m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (89m) 3s ago 101m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (97m) 3s ago 97m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (97m) 2s ago 97m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (87m) 3s ago 101m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (87m) 3s ago 100m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (86m) 3s ago 100m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (85m) 3s ago 100m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:25:41.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (84m) 2s ago 99m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (83m) 2s ago 99m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (82m) 2s ago 98m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (81m) 2s ago 98m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (90m) 2s ago 98m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (81m) 3s ago 96m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (81m) 2s ago 96m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (81m) 3s ago 95m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:25:41.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (81m) 2s ago 95m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:25:42.254 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:25:42.255 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:25:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:25:42.520 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:25:42.521 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[100622]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[100622]: from='client.60305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[100622]: from='client.50343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[100622]: from='client.60317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/944613699' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[102844]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[102844]: from='client.60305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[102844]: from='client.50343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[102844]: from='client.60317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/944613699' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:42.832 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:25:42.832 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:42.832 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:42.832 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:42.832 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:42 smithi196 ceph-mon[76349]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:42 smithi196 ceph-mon[76349]: from='client.60305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:42 smithi196 ceph-mon[76349]: from='client.50343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:42 smithi196 ceph-mon[76349]: from='client.60317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/944613699' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-mon[100622]: from='client.60329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1782149194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-mon[102844]: from='client.60329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1782149194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:43.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:43.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:43.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:43 smithi196 ceph-mon[76349]: from='client.60329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1782149194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:45.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:44 smithi184 ceph-mon[102844]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:45.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:45.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:44 smithi184 ceph-mon[100622]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:45.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:44 smithi196 ceph-mon[76349]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:45.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:46 smithi184 ceph-mon[100622]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:46 smithi184 ceph-mon[102844]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:46 smithi196 ceph-mon[76349]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:48 smithi196 ceph-mon[76349]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:48 smithi184 ceph-mon[100622]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:48 smithi184 ceph-mon[102844]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-mon[100622]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-mon[102844]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:50.479Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:50.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:50.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:50 smithi196 ceph-mon[76349]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:50.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:25:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:25:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:52 smithi184 ceph-mon[100622]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:52 smithi184 ceph-mon[102844]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:52 smithi196 ceph-mon[76349]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:53.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:53.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:25:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:25:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:25:53.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:25:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:54 smithi184 ceph-mon[100622]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:54 smithi184 ceph-mon[102844]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:54 smithi196 ceph-mon[76349]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:57.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:56 smithi184 ceph-mon[100622]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:57.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:56 smithi184 ceph-mon[102844]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:56 smithi196 ceph-mon[76349]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:59.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[100622]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:59.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:59.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[102844]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:59.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.408 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:59 smithi196 ceph-mon[76349]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:25:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:00.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:00.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:00.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:00 smithi196 ceph-mon[76349]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[100622]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[102844]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:26:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:02 smithi196 ceph-mon[76349]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:02 smithi184 ceph-mon[100622]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:02 smithi184 ceph-mon[102844]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:03.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:03.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:03.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:04 smithi196 ceph-mon[76349]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:04 smithi184 ceph-mon[100622]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:04 smithi184 ceph-mon[102844]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:06 smithi196 ceph-mon[76349]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:06 smithi184 ceph-mon[100622]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:06 smithi184 ceph-mon[102844]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:09.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:08 smithi196 ceph-mon[76349]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:08 smithi184 ceph-mon[102844]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:08 smithi184 ceph-mon[100622]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:10.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:10.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:10.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:10 smithi196 ceph-mon[76349]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-mon[100622]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-mon[102844]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:26:13.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:12 smithi184 ceph-mon[100622]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:13.128 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:12 smithi184 ceph-mon[102844]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:13.128 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:26:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:12 smithi196 ceph-mon[76349]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:13.661 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 102m - - 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 102m - - 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (97m) 15s ago 98m 29.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (98m) 13s ago 98m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (91m) 15s ago 96m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (90m) 13s ago 102m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (91m) 15s ago 103m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (90m) 15s ago 103m 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (89m) 13s ago 102m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (89m) 15s ago 102m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (97m) 15s ago 97m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (97m) 13s ago 97m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:26:13.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (88m) 15s ago 101m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (87m) 15s ago 101m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (86m) 15s ago 100m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (86m) 15s ago 100m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (85m) 13s ago 100m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (84m) 13s ago 99m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (83m) 13s ago 99m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (82m) 13s ago 99m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (91m) 13s ago 98m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (82m) 15s ago 96m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (81m) 13s ago 96m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (82m) 15s ago 96m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:26:13.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (81m) 13s ago 96m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:26:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:26:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:26:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:26:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:26:13.976 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:26:14.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:13 smithi196 ceph-mon[76349]: from='client.60341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:14.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:14.245 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:26:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:26:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:26:14.246 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:26:14.247 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-mon[100622]: from='client.60341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:14.247 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:14.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-mon[102844]: from='client.60341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:14.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:14.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:13.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:14.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:13.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:14.247 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:13.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:14.559 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:26:14.559 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:14.559 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:14 smithi196 ceph-mon[76349]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:14 smithi196 ceph-mon[76349]: from='client.60347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:14 smithi196 ceph-mon[76349]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3898942410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1805726774' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[100622]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[100622]: from='client.60347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[100622]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3898942410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1805726774' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[102844]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[102844]: from='client.60347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[102844]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:15.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3898942410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:15.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1805726774' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:15 smithi196 ceph-mon[76349]: from='client.60365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:15 smithi184 ceph-mon[100622]: from='client.60365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:15 smithi184 ceph-mon[102844]: from='client.60365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:16 smithi196 ceph-mon[76349]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:16 smithi184 ceph-mon[100622]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:16 smithi184 ceph-mon[102844]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:18.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[100622]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:18.943 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[102844]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:18.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:18 smithi196 ceph-mon[76349]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:19.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:20.481Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:20.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:20.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[100622]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[102844]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:20 smithi196 ceph-mon[76349]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:26:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:22 smithi184 ceph-mon[100622]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:22 smithi184 ceph-mon[102844]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:22 smithi196 ceph-mon[76349]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:23.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:23.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:23.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:24 smithi184 ceph-mon[100622]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:24 smithi184 ceph-mon[102844]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:24 smithi196 ceph-mon[76349]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:26 smithi184 ceph-mon[100622]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:26 smithi184 ceph-mon[102844]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:26 smithi196 ceph-mon[76349]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:28 smithi184 ceph-mon[100622]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:28 smithi184 ceph-mon[102844]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:29.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:28 smithi196 ceph-mon[76349]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:29.355 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:30.483Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:30.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:30.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-mon[100622]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-mon[102844]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:30 smithi196 ceph-mon[76349]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:31.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:26:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:32 smithi184 ceph-mon[100622]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:32 smithi184 ceph-mon[102844]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:32 smithi196 ceph-mon[76349]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:33.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:33.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:33.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:34 smithi184 ceph-mon[100622]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:34 smithi184 ceph-mon[102844]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:34 smithi196 ceph-mon[76349]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:36 smithi184 ceph-mon[100622]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:36 smithi184 ceph-mon[102844]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:36 smithi196 ceph-mon[76349]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:39.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[100622]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:26:39.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[102844]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:26:39.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:38 smithi196 ceph-mon[76349]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:26:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:40.683 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:40.483Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:40.683 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:40.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:40.683 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:40.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:41 smithi196 ceph-mon[76349]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[102844]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[100622]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:26:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:43 smithi196 ceph-mon[76349]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:43 smithi184 ceph-mon[100622]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:43 smithi184 ceph-mon[102844]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:43.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:43.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:43.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:44.854 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:26:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:45 smithi184 ceph-mon[100622]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:45 smithi184 ceph-mon[102844]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 103m - - 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 103m - - 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (97m) 6s ago 98m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (98m) 5s ago 98m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (91m) 6s ago 96m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (91m) 5s ago 102m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:26:45.387 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (92m) 6s ago 104m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (90m) 6s ago 104m 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (90m) 5s ago 103m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (90m) 6s ago 102m 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (98m) 6s ago 98m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (98m) 5s ago 98m 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (89m) 6s ago 102m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (88m) 6s ago 101m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (87m) 6s ago 101m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (86m) 6s ago 101m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (85m) 5s ago 100m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (84m) 5s ago 100m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (83m) 5s ago 99m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (83m) 5s ago 99m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (91m) 5s ago 99m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:26:45.388 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (82m) 6s ago 97m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:26:45.389 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (82m) 5s ago 97m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:26:45.389 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (82m) 6s ago 96m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:26:45.389 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (82m) 5s ago 96m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:26:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:45 smithi196 ceph-mon[76349]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:26:45.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:26:45.698 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:26:45.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:26:45.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:26:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:46 smithi196 ceph-mon[76349]: from='client.50409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:46 smithi196 ceph-mon[76349]: from='client.50415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4146321760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:46.276 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:26:46.276 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:46.276 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:46.276 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:46.276 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[100622]: from='client.50409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[100622]: from='client.50415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4146321760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[102844]: from='client.50409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[102844]: from='client.50415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:46.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4146321760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:47 smithi196 ceph-mon[76349]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:47 smithi196 ceph-mon[76349]: from='client.50421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:47 smithi196 ceph-mon[76349]: from='client.60401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/215565309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[100622]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[100622]: from='client.50421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[100622]: from='client.60401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/215565309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[102844]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[102844]: from='client.50421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[102844]: from='client.60401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/215565309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:49 smithi196 ceph-mon[76349]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:49 smithi184 ceph-mon[102844]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:49 smithi184 ceph-mon[100622]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:50.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:50.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:50.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:51.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:51 smithi196 ceph-mon[76349]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:51.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:51 smithi184 ceph-mon[102844]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:51 smithi184 ceph-mon[100622]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:26:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:26:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:53 smithi196 ceph-mon[76349]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:53 smithi184 ceph-mon[100622]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:53 smithi184 ceph-mon[102844]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:53.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:53.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:26:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:26:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:26:53.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:26:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:55 smithi196 ceph-mon[76349]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:55 smithi184 ceph-mon[100622]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:55 smithi184 ceph-mon[102844]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:57 smithi196 ceph-mon[76349]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:57 smithi184 ceph-mon[100622]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:57 smithi184 ceph-mon[102844]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:59 smithi196 ceph-mon[76349]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[102844]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[100622]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:26:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:00.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:00.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:00.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[100622]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[102844]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:01 smithi196 ceph-mon[76349]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:27:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:03 smithi196 ceph-mon[76349]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:03 smithi184 ceph-mon[100622]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:03 smithi184 ceph-mon[102844]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:03.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:03.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:03.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:05 smithi196 ceph-mon[76349]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:05 smithi184 ceph-mon[100622]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:05 smithi184 ceph-mon[102844]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:07 smithi196 ceph-mon[76349]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:07 smithi184 ceph-mon[100622]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:07 smithi184 ceph-mon[102844]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:09 smithi196 ceph-mon[76349]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:09 smithi184 ceph-mon[100622]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:09 smithi184 ceph-mon[102844]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:10.486Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:10.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:10.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:11.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:11 smithi196 ceph-mon[76349]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:11.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:11 smithi184 ceph-mon[100622]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:11 smithi184 ceph-mon[102844]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:27:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:13 smithi184 ceph-mon[100622]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:13 smithi184 ceph-mon[102844]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:13 smithi196 ceph-mon[76349]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:13.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:13.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:13.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:15 smithi196 ceph-mon[76349]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:15 smithi184 ceph-mon[100622]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:15 smithi184 ceph-mon[102844]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:16.571 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:27:17.099 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 104m - - 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 103m - - 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (98m) 18s ago 99m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (99m) 16s ago 99m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (92m) 18s ago 97m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (92m) 16s ago 103m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (92m) 18s ago 104m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (91m) 18s ago 104m 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (90m) 16s ago 103m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (90m) 18s ago 103m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (98m) 18s ago 98m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (98m) 16s ago 98m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:27:17.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (89m) 18s ago 102m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (88m) 18s ago 102m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (88m) 18s ago 102m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (87m) 18s ago 101m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (86m) 16s ago 101m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (85m) 16s ago 100m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (84m) 16s ago 100m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (83m) 16s ago 100m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (92m) 16s ago 99m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (83m) 18s ago 97m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (83m) 16s ago 97m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (83m) 18s ago 97m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:27:17.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (82m) 16s ago 97m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:27:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:17 smithi196 ceph-mon[76349]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:17.414 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:27:17.414 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:27:17.414 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:27:17.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:17.414 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:27:17.415 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:27:17.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:17 smithi184 ceph-mon[100622]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:17.416 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:17 smithi184 ceph-mon[102844]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:17.686 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:27:17.687 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:27:17.998 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:27:17.998 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:17.999 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[102844]: from='client.50445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[102844]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[102844]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3038419391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2183427746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[100622]: from='client.50445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[100622]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[100622]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3038419391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2183427746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:18 smithi196 ceph-mon[76349]: from='client.50445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:18 smithi196 ceph-mon[76349]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:18 smithi196 ceph-mon[76349]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3038419391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2183427746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:19.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[100622]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[100622]: from='client.50469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[102844]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:19.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[102844]: from='client.50469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:19.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:19 smithi196 ceph-mon[76349]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:19 smithi196 ceph-mon[76349]: from='client.50469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:20.486Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:20.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:20.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[100622]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[102844]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:21 smithi196 ceph-mon[76349]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:27:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:23 smithi184 ceph-mon[100622]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:23 smithi184 ceph-mon[102844]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:23 smithi196 ceph-mon[76349]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:23.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:23.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:23.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:25 smithi184 ceph-mon[100622]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:25 smithi184 ceph-mon[102844]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:25 smithi196 ceph-mon[76349]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:27.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:27 smithi184 ceph-mon[100622]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:27.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:27 smithi184 ceph-mon[102844]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:27 smithi196 ceph-mon[76349]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:29 smithi184 ceph-mon[102844]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:29 smithi184 ceph-mon[100622]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:29 smithi196 ceph-mon[76349]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:30.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:30.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:30.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:31 smithi184 ceph-mon[100622]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:31 smithi184 ceph-mon[102844]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:31 smithi196 ceph-mon[76349]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:27:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:33 smithi196 ceph-mon[76349]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:33.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:33 smithi184 ceph-mon[100622]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:33.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:33 smithi184 ceph-mon[102844]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:33.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:33.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:33.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:35 smithi196 ceph-mon[76349]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:35 smithi184 ceph-mon[100622]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:35 smithi184 ceph-mon[102844]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:37 smithi184 ceph-mon[100622]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:37 smithi184 ceph-mon[102844]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:37 smithi196 ceph-mon[76349]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[100622]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[102844]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:39 smithi196 ceph-mon[76349]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:40.782 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:40.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:40.782 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:40.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:40.783 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:40.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:41.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[100622]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:41.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:41.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.452 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[102844]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:41 smithi196 ceph-mon[76349]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:41.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:42.232 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:27:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:43 smithi184 ceph-mon[100622]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:43 smithi184 ceph-mon[102844]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:43 smithi196 ceph-mon[76349]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:43.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:43.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:43.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:44.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:44.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:44.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:45 smithi196 ceph-mon[76349]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:45 smithi184 ceph-mon[100622]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:45.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:45 smithi184 ceph-mon[102844]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:46 smithi184 ceph-mon[100622]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:46 smithi184 ceph-mon[102844]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:46 smithi196 ceph-mon[76349]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:48.293 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:27:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:48 smithi184 ceph-mon[100622]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:48 smithi184 ceph-mon[102844]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:48 smithi196 ceph-mon[76349]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 104m - - 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 104m - - 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (98m) 10s ago 99m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (99m) 8s ago 99m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (92m) 10s ago 97m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (92m) 8s ago 103m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (93m) 10s ago 105m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (91m) 10s ago 105m 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:27:48.820 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (91m) 8s ago 104m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (91m) 10s ago 103m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (99m) 10s ago 99m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (99m) 8s ago 99m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (90m) 10s ago 103m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (89m) 10s ago 102m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (88m) 10s ago 102m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (88m) 10s ago 102m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (86m) 8s ago 101m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (85m) 8s ago 101m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (84m) 8s ago 101m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (84m) 8s ago 100m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (92m) 8s ago 100m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (83m) 10s ago 98m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (83m) 8s ago 98m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:27:48.821 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (83m) 10s ago 97m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:27:48.822 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (83m) 8s ago 97m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:27:49.126 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:27:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:27:49.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:27:49.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:27:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:27:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:27:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:27:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:27:49.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[102844]: from='client.50481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[102844]: from='client.60455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[102844]: from='client.60461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2909781349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[100622]: from='client.50481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[100622]: from='client.60455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[100622]: from='client.60461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2909781349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:49 smithi196 ceph-mon[76349]: from='client.50481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:49 smithi196 ceph-mon[76349]: from='client.60455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:49 smithi196 ceph-mon[76349]: from='client.60461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:49.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2909781349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:27:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:49.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:49.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:50 smithi196 ceph-mon[76349]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:50 smithi196 ceph-mon[76349]: from='client.60473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/641235282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:50.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[100622]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[100622]: from='client.60473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/641235282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:50.488Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:50.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:50.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:50.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[102844]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:50.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[102844]: from='client.60473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:50.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/641235282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:50.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:27:52] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T21:27:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:52 smithi196 ceph-mon[76349]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:52 smithi184 ceph-mon[100622]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:52 smithi184 ceph-mon[102844]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:53.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:53.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:27:54.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:27:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:27:53.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:27:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:54 smithi196 ceph-mon[76349]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:54 smithi184 ceph-mon[100622]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:54 smithi184 ceph-mon[102844]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:56 smithi196 ceph-mon[76349]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:56 smithi184 ceph-mon[100622]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:56 smithi184 ceph-mon[102844]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:58 smithi196 ceph-mon[76349]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:58 smithi184 ceph-mon[100622]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:58 smithi184 ceph-mon[102844]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:27:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:00.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:00.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:00.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:00.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:00.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:00.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[100622]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[102844]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:00 smithi196 ceph-mon[76349]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:28:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:02 smithi184 ceph-mon[102844]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:02 smithi184 ceph-mon[100622]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:02 smithi196 ceph-mon[76349]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:03.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:03.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:03.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:05.193 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:04 smithi196 ceph-mon[76349]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:04 smithi184 ceph-mon[100622]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:04 smithi184 ceph-mon[102844]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:06 smithi184 ceph-mon[100622]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:06 smithi184 ceph-mon[102844]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:07.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:06 smithi196 ceph-mon[76349]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:08 smithi184 ceph-mon[100622]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:08 smithi184 ceph-mon[102844]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:08 smithi196 ceph-mon[76349]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:10.490Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:10.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:10.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-mon[100622]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-mon[102844]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:10 smithi196 ceph-mon[76349]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:28:13.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:12 smithi196 ceph-mon[76349]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:28:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:12 smithi184 ceph-mon[100622]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:28:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:12 smithi184 ceph-mon[102844]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:28:13.879 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:28:13.812174199Z level=info msg="Completed cleanup jobs" duration=12.584756ms 2024-09-20T21:28:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:13.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:13.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:13.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:14 smithi184 ceph-mon[100622]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:14 smithi184 ceph-mon[102844]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:14 smithi196 ceph-mon[76349]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:16 smithi184 ceph-mon[100622]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:16 smithi184 ceph-mon[102844]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:16 smithi196 ceph-mon[76349]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:18 smithi184 ceph-mon[100622]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:18 smithi184 ceph-mon[102844]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:18 smithi196 ceph-mon[76349]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:20.006 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:28:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 105m - - 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 104m - - 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (99m) 1s ago 100m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (100m) 19s ago 100m 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (93m) 1s ago 98m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:28:20.538 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (93m) 19s ago 104m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (93m) 1s ago 105m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (92m) 1s ago 106m 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (91m) 19s ago 104m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (92m) 1s ago 104m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (99m) 1s ago 99m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (99m) 19s ago 99m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (90m) 1s ago 103m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (89m) 1s ago 103m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (89m) 1s ago 103m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (88m) 1s ago 102m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (87m) 19s ago 102m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (86m) 19s ago 101m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (85m) 19s ago 101m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:28:20.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (84m) 19s ago 101m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:28:20.540 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (93m) 19s ago 100m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:28:20.540 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (84m) 1s ago 98m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:28:20.540 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (84m) 19s ago 98m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:28:20.540 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (84m) 1s ago 98m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:28:20.540 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (83m) 19s ago 98m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:28:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:20.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:20.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:20.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:20.852 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:28:20.852 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:28:20.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:28:20.853 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:28:21.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:28:21.128 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:28:21.128 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:28:21.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='client.50517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2403931985' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='client.50517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2403931985' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='client.50517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:21.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2403931985' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:21.447 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:28:21.447 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:21.447 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:21.447 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:21.447 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[100622]: from='client.60491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[100622]: from='client.60497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[100622]: from='client.60509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3423099093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[102844]: from='client.60491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[102844]: from='client.60497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[102844]: from='client.60509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3423099093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:22 smithi196 ceph-mon[76349]: from='client.60491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:22 smithi196 ceph-mon[76349]: from='client.60497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:22 smithi196 ceph-mon[76349]: from='client.60509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:22.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3423099093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:23 smithi196 ceph-mon[76349]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:23.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:23 smithi184 ceph-mon[100622]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:23.517 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:23 smithi184 ceph-mon[102844]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:24.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:23.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:24.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:23.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:24.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:23.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:25 smithi196 ceph-mon[76349]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:25 smithi184 ceph-mon[100622]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:25 smithi184 ceph-mon[102844]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:27 smithi196 ceph-mon[76349]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:27 smithi184 ceph-mon[100622]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:27 smithi184 ceph-mon[102844]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:29 smithi196 ceph-mon[76349]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:29 smithi184 ceph-mon[100622]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:29 smithi184 ceph-mon[102844]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:30.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:30.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:30.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:31 smithi196 ceph-mon[76349]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:31 smithi184 ceph-mon[100622]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:31 smithi184 ceph-mon[102844]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:28:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:33 smithi196 ceph-mon[76349]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:33 smithi184 ceph-mon[100622]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:33 smithi184 ceph-mon[102844]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:33.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:33.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:33.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:35.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:35 smithi196 ceph-mon[76349]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:35 smithi184 ceph-mon[100622]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:35 smithi184 ceph-mon[102844]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:37 smithi196 ceph-mon[76349]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:37 smithi184 ceph-mon[100622]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:37 smithi184 ceph-mon[102844]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:39 smithi196 ceph-mon[76349]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:39 smithi184 ceph-mon[100622]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:39 smithi184 ceph-mon[102844]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:40.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:40.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:40.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[100622]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[102844]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:41 smithi196 ceph-mon[76349]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:28:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:43 smithi184 ceph-mon[102844]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:43 smithi184 ceph-mon[100622]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:43 smithi196 ceph-mon[76349]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:44.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:43.928Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:43.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:43.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:44.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:44.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:45 smithi184 ceph-mon[100622]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:45 smithi184 ceph-mon[102844]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:45 smithi196 ceph-mon[76349]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:46 smithi184 ceph-mon[100622]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:46 smithi184 ceph-mon[102844]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:46 smithi196 ceph-mon[76349]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:48 smithi196 ceph-mon[76349]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:48.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:48 smithi184 ceph-mon[100622]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:48.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:48 smithi184 ceph-mon[102844]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:50.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:50 smithi196 ceph-mon[76349]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:50.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-mon[100622]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-mon[102844]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:50.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:50.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:50.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:51.746 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:28:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:28:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 105m - - 2024-09-20T21:28:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 105m - - 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (99m) 13s ago 100m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (100m) 11s ago 100m 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (93m) 13s ago 98m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (93m) 11s ago 104m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (94m) 13s ago 106m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (92m) 13s ago 106m 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (92m) 11s ago 105m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (92m) 13s ago 105m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (100m) 13s ago 100m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (100m) 11s ago 100m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (91m) 13s ago 104m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (90m) 13s ago 104m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (89m) 13s ago 103m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:28:52.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (89m) 13s ago 103m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (87m) 11s ago 102m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (86m) 11s ago 102m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (85m) 11s ago 102m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (85m) 11s ago 101m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (94m) 11s ago 101m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (84m) 13s ago 99m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (84m) 11s ago 99m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (84m) 13s ago 98m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:28:52.272 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (84m) 11s ago 98m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:28:52.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:28:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[102844]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[102844]: from='client.60521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[102844]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[100622]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[100622]: from='client.60521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:52 smithi184 ceph-mon[100622]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:28:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:28:52.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:28:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:52 smithi196 ceph-mon[76349]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:52 smithi196 ceph-mon[76349]: from='client.60521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:52 smithi196 ceph-mon[76349]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:28:52.853 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:28:52.854 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:28:52.854 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:28:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:28:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:53.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:53.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:53 smithi196 ceph-mon[76349]: from='client.50565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3626638639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:53 smithi196 ceph-mon[76349]: from='client.50577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1344492238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[100622]: from='client.50565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3626638639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[100622]: from='client.50577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1344492238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[102844]: from='client.50565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3626638639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[102844]: from='client.50577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1344492238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:53.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:53.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:28:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:28:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:28:53.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:28:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:54 smithi196 ceph-mon[76349]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:54 smithi184 ceph-mon[100622]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:54 smithi184 ceph-mon[102844]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:56 smithi196 ceph-mon[76349]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:56 smithi184 ceph-mon[100622]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:56 smithi184 ceph-mon[102844]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:58 smithi196 ceph-mon[76349]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:58 smithi184 ceph-mon[100622]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:58 smithi184 ceph-mon[102844]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:28:59.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:00 smithi196 ceph-mon[76349]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-mon[100622]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-mon[102844]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:00.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:00.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:00.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:03.205 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:02 smithi184 ceph-mon[100622]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:02 smithi184 ceph-mon[102844]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:02 smithi196 ceph-mon[76349]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:04.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:03.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:04.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:03.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:04.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:03.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:05.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:04 smithi184 ceph-mon[100622]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:05.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:04 smithi184 ceph-mon[102844]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:04 smithi196 ceph-mon[76349]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:06 smithi184 ceph-mon[102844]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:06 smithi184 ceph-mon[100622]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:06 smithi196 ceph-mon[76349]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:08 smithi184 ceph-mon[100622]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:08 smithi184 ceph-mon[102844]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:08 smithi196 ceph-mon[76349]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:10.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:10.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:10.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-mon[100622]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:11.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-mon[102844]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:11.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:10 smithi196 ceph-mon[76349]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:29:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:12 smithi184 ceph-mon[100622]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:12 smithi184 ceph-mon[102844]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:12 smithi196 ceph-mon[76349]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:13.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:13.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:13.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:15 smithi184 ceph-mon[100622]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:15 smithi184 ceph-mon[102844]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:15.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:15 smithi196 ceph-mon[76349]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:17 smithi184 ceph-mon[102844]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:17 smithi184 ceph-mon[100622]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:17 smithi196 ceph-mon[76349]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:19 smithi184 ceph-mon[100622]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:19 smithi184 ceph-mon[102844]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:19 smithi196 ceph-mon[76349]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[100622]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:20.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:20.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:20.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:20.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:20.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[102844]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:20.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:20 smithi196 ceph-mon[76349]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:20.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:23 smithi196 ceph-mon[76349]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:23 smithi184 ceph-mon[100622]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:23 smithi184 ceph-mon[102844]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.472 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 106m - - 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 105m - - 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (100m) 4s ago 101m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (101m) 2s ago 101m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (94m) 4s ago 99m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (94m) 2s ago 105m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (95m) 4s ago 106m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (93m) 4s ago 107m 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (92m) 2s ago 105m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (93m) 4s ago 105m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:29:24.016 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (100m) 4s ago 100m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (100m) 2s ago 100m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (91m) 4s ago 104m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (90m) 4s ago 104m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (90m) 4s ago 104m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (89m) 4s ago 103m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (88m) 2s ago 103m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (87m) 2s ago 103m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (86m) 2s ago 102m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (85m) 2s ago 102m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (94m) 2s ago 101m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (85m) 4s ago 99m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (85m) 2s ago 99m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (85m) 4s ago 99m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:29:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (85m) 2s ago 99m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:29:24.018 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:23.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:24.018 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:23.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:24.018 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:23.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:24.333 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:29:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:29:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:29:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:29:24.334 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:29:24.609 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:29:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:29:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:29:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:29:24.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:29:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:29:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: from='client.60557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: from='client.50595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: from='client.50601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/622081004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:25.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3684674267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: from='client.60557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: from='client.50595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: from='client.50601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/622081004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:25.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3684674267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: from='client.60557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: from='client.50595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: from='client.50601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/622081004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3684674267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:26 smithi196 ceph-mon[76349]: from='client.60581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:26 smithi184 ceph-mon[100622]: from='client.60581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:26 smithi184 ceph-mon[102844]: from='client.60581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:27 smithi196 ceph-mon[76349]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:27 smithi184 ceph-mon[100622]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:27 smithi184 ceph-mon[102844]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:29 smithi196 ceph-mon[76349]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:29 smithi184 ceph-mon[100622]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:29 smithi184 ceph-mon[102844]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:30.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:30.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:30.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:31 smithi196 ceph-mon[76349]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:31 smithi184 ceph-mon[100622]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:31 smithi184 ceph-mon[102844]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:29:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:33 smithi196 ceph-mon[76349]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:33.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:33 smithi184 ceph-mon[100622]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:33.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:33 smithi184 ceph-mon[102844]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:33.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:33.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:33.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:35.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:35 smithi196 ceph-mon[76349]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:35 smithi184 ceph-mon[102844]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:35 smithi184 ceph-mon[100622]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:37 smithi196 ceph-mon[76349]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:37 smithi184 ceph-mon[102844]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:37 smithi184 ceph-mon[100622]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:39 smithi196 ceph-mon[76349]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:39 smithi184 ceph-mon[100622]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:39 smithi184 ceph-mon[102844]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[100622]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[102844]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:40.496Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:40.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:40.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:40.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:40 smithi196 ceph-mon[76349]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:40.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[100622]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:29:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:42 smithi184 ceph-mon[102844]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:29:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:42 smithi196 ceph-mon[76349]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:44.198 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:43.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:44.198 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:43.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:44.198 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:43.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:44.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:44 smithi184 ceph-mon[100622]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:44.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:44.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:44 smithi184 ceph-mon[102844]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:44.536 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:44 smithi196 ceph-mon[76349]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:46 smithi184 ceph-mon[100622]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:46 smithi184 ceph-mon[102844]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:46 smithi196 ceph-mon[76349]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:48 smithi196 ceph-mon[76349]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:48 smithi184 ceph-mon[100622]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:48 smithi184 ceph-mon[102844]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:50 smithi196 ceph-mon[76349]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-mon[100622]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-mon[102844]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:50.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:50.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:50.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:29:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:29:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:52 smithi196 ceph-mon[76349]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:52 smithi184 ceph-mon[100622]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:52 smithi184 ceph-mon[102844]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:53.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:53.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:29:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:29:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:29:53.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:29:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:54 smithi196 ceph-mon[76349]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:54 smithi184 ceph-mon[100622]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:54 smithi184 ceph-mon[102844]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:55.250 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 106m - - 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 106m - - 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (100m) 16s ago 101m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (101m) 14s ago 101m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (94m) 16s ago 99m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (94m) 14s ago 105m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (95m) 16s ago 107m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (93m) 16s ago 107m 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (93m) 14s ago 106m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (93m) 16s ago 106m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:29:55.780 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (101m) 16s ago 101m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (101m) 14s ago 101m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (92m) 16s ago 105m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (91m) 16s ago 105m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (90m) 16s ago 104m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (90m) 16s ago 104m 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (88m) 14s ago 103m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (88m) 14s ago 103m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (86m) 14s ago 103m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (86m) 14s ago 102m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (95m) 14s ago 102m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (85m) 16s ago 100m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (85m) 14s ago 100m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (85m) 16s ago 100m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:29:55.781 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (85m) 14s ago 99m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:29:56.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:29:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:29:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:29:56.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:29:56.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:29:56.359 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:29:56.359 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:29:56.359 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:29:56.360 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:29:56.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:56 smithi196 ceph-mon[76349]: from='client.50625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:56 smithi196 ceph-mon[76349]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:56.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:56 smithi196 ceph-mon[76349]: from='client.60599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:56 smithi196 ceph-mon[76349]: from='client.60605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/350551067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:29:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:56.670 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:56.670 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[102844]: from='client.50625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[102844]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[102844]: from='client.60599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[102844]: from='client.60605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/350551067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:56.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[100622]: from='client.50625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[100622]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[100622]: from='client.60599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[100622]: from='client.60605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/350551067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:57.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:57 smithi196 ceph-mon[76349]: from='client.60617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:57.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/229084570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:57 smithi184 ceph-mon[102844]: from='client.60617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/229084570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:57 smithi184 ceph-mon[100622]: from='client.60617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/229084570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:58 smithi184 ceph-mon[100622]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:29:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:58 smithi184 ceph-mon[102844]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:29:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:58 smithi196 ceph-mon[76349]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:29:59.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:59.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:00.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:00.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:00.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:30:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:02 smithi196 ceph-mon[76349]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[100622]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[102844]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:02.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:02.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:03.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:03.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:03.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:04 smithi196 ceph-mon[76349]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:04.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:04 smithi184 ceph-mon[100622]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:04.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:04 smithi184 ceph-mon[102844]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:30:06.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:06 smithi184 ceph-mon[100622]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:06.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:06 smithi184 ceph-mon[102844]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:06 smithi196 ceph-mon[76349]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:08 smithi196 ceph-mon[76349]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:08 smithi184 ceph-mon[100622]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:08 smithi184 ceph-mon[102844]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:10 smithi196 ceph-mon[76349]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-mon[100622]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-mon[102844]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:10.499Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:10.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:10.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:10.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:30:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:12 smithi196 ceph-mon[76349]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:12 smithi184 ceph-mon[100622]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:12 smithi184 ceph-mon[102844]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:13.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:13.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:13.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:14 smithi196 ceph-mon[76349]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:14.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:14 smithi184 ceph-mon[100622]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:14 smithi184 ceph-mon[102844]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:16 smithi196 ceph-mon[76349]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:16 smithi184 ceph-mon[100622]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:16 smithi184 ceph-mon[102844]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:18 smithi196 ceph-mon[76349]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:18 smithi184 ceph-mon[100622]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:18 smithi184 ceph-mon[102844]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:20.500Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:20.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:20.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:20 smithi196 ceph-mon[76349]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[102844]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[100622]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:21.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[102844]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[100622]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:22 smithi196 ceph-mon[76349]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:22.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:23.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:23.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:23.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:24 smithi184 ceph-mon[100622]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:24 smithi184 ceph-mon[102844]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:24 smithi196 ceph-mon[76349]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:26.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:26 smithi184 ceph-mon[100622]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:26.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:26 smithi184 ceph-mon[102844]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:26 smithi196 ceph-mon[76349]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:26.963 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:30:27.567 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 107m - - 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 106m - - 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (101m) 7s ago 102m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (102m) 6s ago 102m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (95m) 7s ago 100m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (95m) 6s ago 106m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (96m) 7s ago 108m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (94m) 7s ago 108m 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (93m) 6s ago 106m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (94m) 7s ago 106m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (102m) 7s ago 102m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:30:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (101m) 6s ago 101m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (92m) 7s ago 105m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (91m) 7s ago 105m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (91m) 7s ago 105m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (90m) 7s ago 104m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (89m) 6s ago 104m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (88m) 6s ago 104m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (87m) 6s ago 103m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (86m) 6s ago 103m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (95m) 6s ago 102m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (86m) 7s ago 100m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (86m) 6s ago 101m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (86m) 7s ago 100m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:30:27.569 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (86m) 6s ago 100m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:30:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:27 smithi196 ceph-mon[76349]: from='client.60629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:27.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:27 smithi184 ceph-mon[100622]: from='client.60629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:27.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:27 smithi184 ceph-mon[102844]: from='client.60629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:30:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:30:27.895 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:30:28.186 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:30:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:30:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:30:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:30:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:30:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:30:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:30:28.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:30:28.504 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:28.504 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:28.504 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: from='client.60641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3733032512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: from='client.60653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1619848561' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: from='client.60641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3733032512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: from='client.60653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1619848561' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: from='client.60641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3733032512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: from='client.60653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:28.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1619848561' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-mon[100622]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-mon[102844]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:30.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:30.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:30.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:30 smithi196 ceph-mon[76349]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:30:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:32 smithi196 ceph-mon[76349]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:32 smithi184 ceph-mon[102844]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:32 smithi184 ceph-mon[100622]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:33.936Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:33.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:33.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:34 smithi196 ceph-mon[76349]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:34 smithi184 ceph-mon[100622]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:34 smithi184 ceph-mon[102844]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:36.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:36 smithi196 ceph-mon[76349]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:36 smithi184 ceph-mon[100622]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:36 smithi184 ceph-mon[102844]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:38 smithi196 ceph-mon[76349]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:38 smithi184 ceph-mon[100622]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:38 smithi184 ceph-mon[102844]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:40.502Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:40.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:40.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[100622]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[102844]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:40 smithi196 ceph-mon[76349]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:42 smithi196 ceph-mon[76349]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[100622]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[102844]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:30:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:43.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:43.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:43.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:44 smithi196 ceph-mon[76349]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:44 smithi184 ceph-mon[100622]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:44 smithi184 ceph-mon[102844]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:46.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:46 smithi184 ceph-mon[100622]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:46.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:46 smithi184 ceph-mon[102844]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:46 smithi196 ceph-mon[76349]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:48.743 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:48 smithi184 ceph-mon[100622]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:48.743 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:48 smithi184 ceph-mon[102844]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:48 smithi196 ceph-mon[76349]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-mon[100622]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-mon[102844]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:50.503Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:50.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:50.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:50.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:50 smithi196 ceph-mon[76349]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:30:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:30:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:52 smithi196 ceph-mon[76349]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:52 smithi184 ceph-mon[100622]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:52 smithi184 ceph-mon[102844]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:53.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:53.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:30:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:30:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:30:53.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:30:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:54 smithi196 ceph-mon[76349]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:54 smithi184 ceph-mon[100622]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:54 smithi184 ceph-mon[102844]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:56 smithi196 ceph-mon[76349]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:56 smithi184 ceph-mon[100622]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:56 smithi184 ceph-mon[102844]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:58.805 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:58 smithi196 ceph-mon[76349]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:58 smithi184 ceph-mon[100622]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:58 smithi184 ceph-mon[102844]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 107m - - 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 107m - - 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (101m) 19s ago 102m 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (102m) 17s ago 102m 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (95m) 19s ago 100m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:30:59.335 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (95m) 17s ago 106m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (96m) 19s ago 108m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (94m) 19s ago 108m 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (94m) 17s ago 107m 143M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (94m) 19s ago 107m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (102m) 19s ago 102m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (102m) 17s ago 102m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (93m) 19s ago 106m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (92m) 19s ago 106m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (91m) 19s ago 105m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (91m) 19s ago 105m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (89m) 17s ago 104m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (89m) 17s ago 104m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (87m) 17s ago 104m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:30:59.336 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (87m) 17s ago 103m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:30:59.337 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (96m) 17s ago 103m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:30:59.337 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (86m) 19s ago 101m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:30:59.337 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (86m) 17s ago 101m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:30:59.337 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (86m) 19s ago 101m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:30:59.337 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (86m) 17s ago 101m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:30:59.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:30:59.652 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:30:59.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:59 smithi196 ceph-mon[76349]: from='client.60665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:59.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:30:59 smithi196 ceph-mon[76349]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:30:59.919 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:30:59.920 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:30:59.920 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:30:59.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:30:59.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[100622]: from='client.60665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:59.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[100622]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:59.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[102844]: from='client.60665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:59.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:30:59 smithi184 ceph-mon[102844]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:00.240 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:31:00.240 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:00.241 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:00.241 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:00.241 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='client.60677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/670829399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='client.50721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1585940648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='client.60677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/670829399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='client.50721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:00.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1585940648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:00.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:00.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:00.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:00.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='client.60677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/670829399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='client.50721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1585940648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:31:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[100622]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[102844]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:02 smithi196 ceph-mon[76349]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:03.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:03.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:03.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:04 smithi184 ceph-mon[100622]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:04 smithi184 ceph-mon[102844]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:04 smithi196 ceph-mon[76349]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:07.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:06 smithi184 ceph-mon[100622]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:07.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:06 smithi184 ceph-mon[102844]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:06 smithi196 ceph-mon[76349]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:08 smithi184 ceph-mon[100622]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:08 smithi184 ceph-mon[102844]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:08 smithi196 ceph-mon[76349]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:10.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-mon[100622]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:10.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:10.756 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:10.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:10.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:10.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:10.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:10.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-mon[102844]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:10 smithi196 ceph-mon[76349]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:31:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:12 smithi184 ceph-mon[100622]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:12 smithi184 ceph-mon[102844]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:12 smithi196 ceph-mon[76349]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:13.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:13.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:13.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:14 smithi184 ceph-mon[100622]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:14 smithi184 ceph-mon[102844]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:14 smithi196 ceph-mon[76349]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:16 smithi184 ceph-mon[100622]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:16 smithi184 ceph-mon[102844]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:16 smithi196 ceph-mon[76349]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:18 smithi184 ceph-mon[100622]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:18 smithi184 ceph-mon[102844]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:19.057 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:18 smithi196 ceph-mon[76349]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:20.505Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:20.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:20.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[100622]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[102844]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:21 smithi196 ceph-mon[76349]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:31:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:22 smithi196 ceph-mon[76349]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[102844]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[100622]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:23.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:23.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:23.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:23.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:24 smithi196 ceph-mon[76349]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:24 smithi184 ceph-mon[100622]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:24 smithi184 ceph-mon[102844]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:26 smithi196 ceph-mon[76349]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:26 smithi184 ceph-mon[100622]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:26 smithi184 ceph-mon[102844]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:28 smithi196 ceph-mon[76349]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:28 smithi184 ceph-mon[100622]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:28 smithi184 ceph-mon[102844]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:30.535 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:31:30.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:30.506Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:30.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:30.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:31:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 108m - - 2024-09-20T21:31:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 107m - - 2024-09-20T21:31:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (102m) 10s ago 103m 28.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (103m) 9s ago 103m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (96m) 10s ago 101m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (96m) 9s ago 107m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (97m) 10s ago 109m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (95m) 10s ago 109m 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (94m) 9s ago 107m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (95m) 10s ago 107m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (103m) 10s ago 103m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (103m) 9s ago 103m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (93m) 10s ago 107m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:31:31.068 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (92m) 10s ago 106m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (92m) 10s ago 106m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (91m) 10s ago 105m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (90m) 9s ago 105m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (89m) 9s ago 105m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (88m) 9s ago 104m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (87m) 9s ago 104m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (96m) 9s ago 103m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (87m) 10s ago 101m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (87m) 9s ago 102m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (87m) 10s ago 101m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:31:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (87m) 9s ago 101m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:31:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:30 smithi196 ceph-mon[76349]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-mon[100622]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-mon[102844]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:31.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:31:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:31:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:31:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:31:31.381 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:31:31.652 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:31:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:31:31.654 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:31:31.966 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:31:31.966 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:31.966 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:31.966 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:31.966 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[102844]: from='client.50733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[102844]: from='client.60707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[102844]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/889853747' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[100622]: from='client.50733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[100622]: from='client.60707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[100622]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:31.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/889853747' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:31 smithi196 ceph-mon[76349]: from='client.50733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:31 smithi196 ceph-mon[76349]: from='client.60707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:31 smithi196 ceph-mon[76349]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/889853747' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:31:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[102844]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[102844]: from='client.50757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3968891105' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[100622]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[100622]: from='client.50757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3968891105' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:32 smithi196 ceph-mon[76349]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:32 smithi196 ceph-mon[76349]: from='client.50757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3968891105' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:33.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:33.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:33.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:34 smithi196 ceph-mon[76349]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:34 smithi184 ceph-mon[100622]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:34 smithi184 ceph-mon[102844]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:36 smithi196 ceph-mon[76349]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:36 smithi184 ceph-mon[100622]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:36 smithi184 ceph-mon[102844]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:39.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:38 smithi196 ceph-mon[76349]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:38 smithi184 ceph-mon[100622]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:38 smithi184 ceph-mon[102844]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:40.507Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:40.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:40.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:41.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:40 smithi196 ceph-mon[76349]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:41.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:41.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[102844]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[100622]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:41.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[102844]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[100622]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:42 smithi196 ceph-mon[76349]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:31:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:43.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:43.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:43.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:44 smithi184 ceph-mon[100622]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:44 smithi184 ceph-mon[102844]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:44 smithi196 ceph-mon[76349]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:46 smithi184 ceph-mon[100622]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:46 smithi184 ceph-mon[102844]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:46 smithi196 ceph-mon[76349]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:49.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:49 smithi184 ceph-mon[100622]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:31:49.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:49 smithi184 ceph-mon[102844]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:31:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:49 smithi196 ceph-mon[76349]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:31:50.702 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:50.507Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:50.702 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:50.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:50.702 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:50.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:51 smithi184 ceph-mon[100622]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:51 smithi184 ceph-mon[102844]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:51 smithi196 ceph-mon[76349]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:31:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:31:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:53 smithi184 ceph-mon[100622]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:53 smithi184 ceph-mon[102844]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:53 smithi196 ceph-mon[76349]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:53.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:53.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:31:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:31:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:31:53.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:31:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:55 smithi184 ceph-mon[100622]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:55 smithi184 ceph-mon[102844]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:55 smithi196 ceph-mon[76349]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:57 smithi184 ceph-mon[100622]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:57 smithi184 ceph-mon[102844]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:57 smithi196 ceph-mon[76349]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:59 smithi184 ceph-mon[100622]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:59 smithi184 ceph-mon[102844]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:59 smithi196 ceph-mon[76349]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:00.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:00.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:00.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:01 smithi196 ceph-mon[76349]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[100622]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[102844]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:02.270 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:32:02.271 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:32:02.795 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:32:02.795 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 108m - - 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 108m - - 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (102m) 2s ago 104m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (103m) 0s ago 103m 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (96m) 2s ago 102m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (96m) 0s ago 107m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (97m) 2s ago 109m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (96m) 2s ago 109m 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (95m) 0s ago 108m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (95m) 2s ago 108m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (103m) 2s ago 103m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (103m) 0s ago 103m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (94m) 2s ago 107m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:32:02.796 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (93m) 2s ago 107m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (92m) 2s ago 106m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (92m) 2s ago 106m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (90m) 0s ago 106m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (90m) 0s ago 105m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (89m) 0s ago 105m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (88m) 0s ago 104m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (97m) 0s ago 104m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (87m) 2s ago 102m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (87m) 0s ago 102m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:32:02.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (87m) 2s ago 102m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:32:02.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (87m) 0s ago 102m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:32:03.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:32:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:32:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:32:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:32:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:32:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:32:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:32:03.108 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:32:03.382 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:32:03.383 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:32:03.383 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:32:03.383 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[100622]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3561470308' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:03.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[102844]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.384 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.384 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.384 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.384 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3561470308' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:03 smithi196 ceph-mon[76349]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3561470308' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:03.695 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:32:03.695 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:03.696 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:03.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:03.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:03.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:04 smithi196 ceph-mon[76349]: from='client.60737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:04 smithi196 ceph-mon[76349]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:04 smithi196 ceph-mon[76349]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2249909514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[100622]: from='client.60737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[100622]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[100622]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2249909514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[102844]: from='client.60737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[102844]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[102844]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2249909514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:05 smithi196 ceph-mon[76349]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:05 smithi196 ceph-mon[76349]: from='client.50793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:05 smithi184 ceph-mon[100622]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:05 smithi184 ceph-mon[100622]: from='client.50793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:05 smithi184 ceph-mon[102844]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:05 smithi184 ceph-mon[102844]: from='client.50793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:07 smithi196 ceph-mon[76349]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:07 smithi184 ceph-mon[100622]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:07 smithi184 ceph-mon[102844]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:09 smithi196 ceph-mon[76349]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:09 smithi184 ceph-mon[100622]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:09 smithi184 ceph-mon[102844]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:10.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:10.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:10.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:11.381 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:11 smithi184 ceph-mon[100622]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:11.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:11.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:11 smithi184 ceph-mon[102844]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:11.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:11 smithi196 ceph-mon[76349]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:32:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:13 smithi196 ceph-mon[76349]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:13 smithi184 ceph-mon[100622]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:13 smithi184 ceph-mon[102844]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:13.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:13.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:13.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:15 smithi184 ceph-mon[100622]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:15 smithi184 ceph-mon[102844]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:15 smithi196 ceph-mon[76349]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:17 smithi184 ceph-mon[100622]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:17 smithi184 ceph-mon[102844]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:17 smithi196 ceph-mon[76349]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:19 smithi184 ceph-mon[100622]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:19 smithi184 ceph-mon[102844]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:19 smithi196 ceph-mon[76349]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:20.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:20.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:20.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:21 smithi196 ceph-mon[76349]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:21.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[100622]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[102844]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[100622]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[102844]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:23 smithi196 ceph-mon[76349]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:23.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:23.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:23.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:25 smithi184 ceph-mon[100622]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:25 smithi184 ceph-mon[102844]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:25 smithi196 ceph-mon[76349]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:27 smithi184 ceph-mon[100622]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:27 smithi184 ceph-mon[102844]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:27 smithi196 ceph-mon[76349]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:29.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:29 smithi184 ceph-mon[100622]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:29.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:29.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:29 smithi184 ceph-mon[102844]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:29.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:29 smithi196 ceph-mon[76349]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:30.764 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:30.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:30.764 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:30.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:30.764 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:30.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:31.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:31 smithi184 ceph-mon[100622]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:31.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:31.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:31 smithi184 ceph-mon[102844]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:31.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:31 smithi196 ceph-mon[76349]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:32:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:33 smithi184 ceph-mon[100622]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:33 smithi184 ceph-mon[102844]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:33 smithi196 ceph-mon[76349]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:32:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:33.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:33.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:33.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 109m - - 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 109m - - 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (103m) 13s ago 104m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (104m) 12s ago 104m 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (97m) 13s ago 102m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (97m) 12s ago 108m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (98m) 13s ago 110m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (96m) 13s ago 110m 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:32:34.521 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (95m) 12s ago 108m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (96m) 13s ago 108m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (104m) 13s ago 104m 15.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (104m) 12s ago 104m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (94m) 13s ago 108m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (93m) 13s ago 107m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (93m) 13s ago 107m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (92m) 13s ago 106m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (91m) 12s ago 106m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (90m) 12s ago 106m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (89m) 12s ago 105m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (88m) 12s ago 105m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (97m) 12s ago 104m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (88m) 13s ago 103m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:32:34.522 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (88m) 12s ago 103m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:32:34.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (88m) 13s ago 102m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:32:34.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (88m) 12s ago 102m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:32:34.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:32:34.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:32:34.834 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:32:34.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:32:34.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:32:34.834 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:32:35.103 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:32:35.104 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:32:35.104 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:32:35.104 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:32:35.104 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:32:35.104 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:32:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:32:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:35.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[102844]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:35.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[102844]: from='client.60773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:35.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2568791500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:35.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[100622]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:35.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[100622]: from='client.60773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:35.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2568791500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:35 smithi196 ceph-mon[76349]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:35 smithi196 ceph-mon[76349]: from='client.60773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2568791500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[102844]: from='client.50811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[102844]: from='client.60785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[102844]: from='client.60797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4218180408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[100622]: from='client.50811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[100622]: from='client.60785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[100622]: from='client.60797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4218180408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:36 smithi196 ceph-mon[76349]: from='client.50811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:36 smithi196 ceph-mon[76349]: from='client.60785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:36 smithi196 ceph-mon[76349]: from='client.60797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:36.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4218180408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:37.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:37 smithi196 ceph-mon[76349]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:37 smithi184 ceph-mon[102844]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:37 smithi184 ceph-mon[100622]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:39 smithi184 ceph-mon[100622]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:39 smithi184 ceph-mon[102844]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:39 smithi196 ceph-mon[76349]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:40.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:40.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:40.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:41 smithi196 ceph-mon[76349]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[100622]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[102844]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:32:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:43 smithi196 ceph-mon[76349]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[100622]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[102844]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:32:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:43.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:43.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:43.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:45 smithi184 ceph-mon[100622]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:45 smithi184 ceph-mon[102844]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:45 smithi196 ceph-mon[76349]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:47 smithi184 ceph-mon[100622]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:47 smithi184 ceph-mon[102844]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:47 smithi196 ceph-mon[76349]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:49 smithi184 ceph-mon[100622]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:49 smithi184 ceph-mon[102844]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:49 smithi196 ceph-mon[76349]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:50.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:50.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:50.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:51.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:51 smithi184 ceph-mon[102844]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:51.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:51 smithi184 ceph-mon[100622]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:51.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:51.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:51 smithi196 ceph-mon[76349]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:52.214 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:32:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:32:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:53 smithi184 ceph-mon[100622]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:53 smithi184 ceph-mon[102844]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:53 smithi196 ceph-mon[76349]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:53.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:53.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:32:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:32:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:32:53.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:32:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:55 smithi184 ceph-mon[100622]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:55 smithi184 ceph-mon[102844]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:55 smithi196 ceph-mon[76349]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:57 smithi184 ceph-mon[100622]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:57 smithi184 ceph-mon[102844]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:57 smithi196 ceph-mon[76349]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:59 smithi184 ceph-mon[100622]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:32:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:59 smithi184 ceph-mon[102844]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:32:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:59 smithi196 ceph-mon[76349]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:32:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:00.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:00.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:00.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:01.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:01 smithi196 ceph-mon[76349]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:01.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[100622]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[102844]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:33:02.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:03.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:03 smithi196 ceph-mon[76349]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[100622]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[102844]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:03.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:03.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:03.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:05 smithi184 ceph-mon[100622]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:05 smithi184 ceph-mon[102844]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:05 smithi196 ceph-mon[76349]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:05.715 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:33:06.243 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:33:06.243 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 109m - - 2024-09-20T21:33:06.243 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 109m - - 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (104m) 5s ago 105m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (104m) 3s ago 104m 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (98m) 5s ago 103m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (97m) 3s ago 108m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (98m) 5s ago 110m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (97m) 5s ago 110m 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (96m) 3s ago 109m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (96m) 5s ago 109m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (104m) 5s ago 104m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (104m) 3s ago 104m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (95m) 5s ago 108m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (94m) 5s ago 108m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (93m) 5s ago 107m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (93m) 5s ago 107m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:33:06.244 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (92m) 3s ago 107m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (91m) 3s ago 106m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (90m) 3s ago 106m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (89m) 3s ago 105m 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (98m) 3s ago 105m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (89m) 5s ago 103m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (88m) 3s ago 103m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (88m) 5s ago 103m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:33:06.245 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (88m) 3s ago 103m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:33:06.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:33:06.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:06.558 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:33:06.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:33:06.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:33:06.558 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:33:06.825 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:33:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:33:06.827 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:33:06.827 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:33:06.827 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:33:07.136 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:33:07.136 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:07.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:07.136 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:07.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[102844]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[102844]: from='client.60809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[102844]: from='client.60815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2064018439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/492723135' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[100622]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[100622]: from='client.60809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[100622]: from='client.60815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2064018439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/492723135' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:07 smithi196 ceph-mon[76349]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:07 smithi196 ceph-mon[76349]: from='client.60809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:07 smithi196 ceph-mon[76349]: from='client.60815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2064018439' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/492723135' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:08 smithi184 ceph-mon[100622]: from='client.60821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:08 smithi184 ceph-mon[100622]: from='client.50865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:08 smithi184 ceph-mon[102844]: from='client.60821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:08 smithi184 ceph-mon[102844]: from='client.50865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:08 smithi196 ceph-mon[76349]: from='client.60821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:08 smithi196 ceph-mon[76349]: from='client.50865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:09 smithi184 ceph-mon[100622]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:09 smithi184 ceph-mon[102844]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:09 smithi196 ceph-mon[76349]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:10.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:10 smithi184 ceph-mon[100622]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:10.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:10 smithi184 ceph-mon[102844]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:10 smithi196 ceph-mon[76349]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:10.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:10.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:10.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:10.801 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:10.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:12.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:33:12.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:12 smithi184 ceph-mon[100622]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:12.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:12 smithi184 ceph-mon[102844]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:12 smithi196 ceph-mon[76349]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:13.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:13.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:13.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:14 smithi184 ceph-mon[100622]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:14 smithi184 ceph-mon[102844]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:14 smithi196 ceph-mon[76349]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:16 smithi184 ceph-mon[100622]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:16 smithi184 ceph-mon[102844]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:16 smithi196 ceph-mon[76349]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:18 smithi184 ceph-mon[100622]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:18 smithi184 ceph-mon[102844]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:18 smithi196 ceph-mon[76349]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:20.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:20 smithi184 ceph-mon[100622]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:20.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:20 smithi184 ceph-mon[102844]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:20.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:20.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:20.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:20 smithi196 ceph-mon[76349]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:22.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:22.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:33:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:22 smithi196 ceph-mon[76349]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[100622]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[102844]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:23.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:23.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:23.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:24 smithi184 ceph-mon[100622]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:25.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:24 smithi184 ceph-mon[102844]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:25.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:24 smithi196 ceph-mon[76349]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:26 smithi184 ceph-mon[100622]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:26 smithi184 ceph-mon[102844]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:26 smithi196 ceph-mon[76349]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:28 smithi184 ceph-mon[100622]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:28 smithi184 ceph-mon[102844]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:28 smithi196 ceph-mon[76349]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:30.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:30.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:30.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:30 smithi184 ceph-mon[100622]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:30 smithi184 ceph-mon[102844]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:30 smithi196 ceph-mon[76349]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:33:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:33.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:32 smithi184 ceph-mon[100622]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:32 smithi184 ceph-mon[102844]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:32 smithi196 ceph-mon[76349]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:34.197 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:33.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:34.197 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:33.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:34.197 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:33.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:34 smithi184 ceph-mon[100622]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:34 smithi184 ceph-mon[102844]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:34 smithi196 ceph-mon[76349]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:36 smithi184 ceph-mon[100622]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:36 smithi184 ceph-mon[102844]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:36 smithi196 ceph-mon[76349]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:37.431 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 110m - - 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 110m - - 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (104m) 17s ago 105m 29.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (105m) 15s ago 105m 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (98m) 17s ago 103m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (98m) 15s ago 109m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:33:37.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (99m) 17s ago 111m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (97m) 17s ago 111m 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (97m) 15s ago 109m 143M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (97m) 17s ago 109m 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (105m) 17s ago 105m 15.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (105m) 15s ago 105m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (95m) 17s ago 109m 70.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (95m) 17s ago 108m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (94m) 17s ago 108m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (93m) 17s ago 107m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (92m) 15s ago 107m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (91m) 15s ago 107m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:33:37.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (90m) 15s ago 106m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (89m) 15s ago 106m 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (98m) 15s ago 105m 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (89m) 17s ago 104m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (89m) 15s ago 104m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (89m) 17s ago 103m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:33:37.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (89m) 15s ago 103m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:33:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:33:38.270 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:33:38.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:33:38.536 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:33:38.847 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:33:38.847 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:38.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:38.848 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:38.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: from='client.60845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: from='client.60851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: from='client.60857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2595178548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2386161955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: from='client.60845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: from='client.60851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: from='client.60857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2595178548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:39.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2386161955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: from='client.60845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: from='client.60851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: from='client.60857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2595178548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2386161955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:40.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:39 smithi184 ceph-mon[100622]: from='client.60869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:40.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:39 smithi184 ceph-mon[102844]: from='client.60869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:39 smithi196 ceph-mon[76349]: from='client.60869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:40.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:40.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:40.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:40 smithi184 ceph-mon[100622]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:40 smithi184 ceph-mon[102844]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:40 smithi196 ceph-mon[76349]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:43 smithi196 ceph-mon[76349]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[100622]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[102844]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:33:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:43.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:43.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:43.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:45 smithi196 ceph-mon[76349]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:45 smithi184 ceph-mon[100622]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:45 smithi184 ceph-mon[102844]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:47 smithi196 ceph-mon[76349]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:47 smithi184 ceph-mon[100622]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:47 smithi184 ceph-mon[102844]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:49 smithi196 ceph-mon[76349]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:49 smithi184 ceph-mon[100622]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:49 smithi184 ceph-mon[102844]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:50.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:50.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:50.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:50.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:51 smithi196 ceph-mon[76349]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:51 smithi184 ceph-mon[100622]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:51 smithi184 ceph-mon[102844]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:33:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:33:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:53.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:53 smithi184 ceph-mon[100622]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:53 smithi184 ceph-mon[102844]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:53 smithi196 ceph-mon[76349]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:53.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:53.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:33:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:33:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:33:53.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:33:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:55 smithi196 ceph-mon[76349]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:55 smithi184 ceph-mon[100622]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:55 smithi184 ceph-mon[102844]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:57 smithi196 ceph-mon[76349]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:57 smithi184 ceph-mon[100622]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:57 smithi184 ceph-mon[102844]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:59 smithi196 ceph-mon[76349]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:59 smithi184 ceph-mon[100622]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:59 smithi184 ceph-mon[102844]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:00.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:00.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:00.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:00.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:01 smithi196 ceph-mon[76349]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:01 smithi184 ceph-mon[100622]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:01 smithi184 ceph-mon[102844]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[100622]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[102844]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:03 smithi196 ceph-mon[76349]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:03.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:03.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:03.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:05.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:05 smithi196 ceph-mon[76349]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:05 smithi184 ceph-mon[100622]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:05 smithi184 ceph-mon[102844]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:07 smithi184 ceph-mon[100622]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:34:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:07 smithi184 ceph-mon[102844]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:34:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:07 smithi196 ceph-mon[76349]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:34:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:34:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:09 smithi184 ceph-mon[100622]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T21:34:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:09 smithi184 ceph-mon[102844]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T21:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:09 smithi196 ceph-mon[76349]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 110m - - 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 110m - - 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (105m) 8s ago 106m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (105m) 6s ago 105m 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (99m) 8s ago 104m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (98m) 6s ago 110m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (99m) 8s ago 111m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (98m) 8s ago 111m 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:34:09.676 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (97m) 6s ago 110m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (97m) 8s ago 110m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (105m) 8s ago 105m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (105m) 6s ago 105m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (96m) 8s ago 109m 70.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (95m) 8s ago 109m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (94m) 8s ago 108m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (94m) 8s ago 108m 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (93m) 6s ago 108m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (92m) 6s ago 107m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (91m) 6s ago 107m 69.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (90m) 6s ago 107m 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (99m) 6s ago 106m 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (90m) 8s ago 104m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:34:09.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (89m) 6s ago 104m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:34:09.678 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (89m) 8s ago 104m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:34:09.678 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (89m) 6s ago 104m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:34:09.987 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:09.988 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:34:09.989 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:34:09.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:34:09.989 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:34:10.256 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:34:10.256 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:34:10.256 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:34:10.256 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:34:10.256 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:34:10.257 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:34:10.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-mon[100622]: from='client.50913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:10.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3222692318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:10.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-mon[102844]: from='client.50913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:10.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3222692318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:10.569 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:34:10.569 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:10.569 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:10.569 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:10 smithi196 ceph-mon[76349]: from='client.50913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3222692318' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:10.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:10.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:10.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: from='client.60887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: from='client.50937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1420791169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: from='client.60887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: from='client.50937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:11.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1420791169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: from='client.60887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-20T21:34:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: from='client.50937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:11.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1420791169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:12.254 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:12 smithi184 ceph-mon[100622]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:12.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:34:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:12 smithi184 ceph-mon[102844]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:12 smithi196 ceph-mon[76349]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:14.205 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:13.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:14.205 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:13.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:14.205 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:13.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:14 smithi184 ceph-mon[100622]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:14.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:14 smithi184 ceph-mon[102844]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:14.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:14 smithi196 ceph-mon[76349]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:34:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:16 smithi184 ceph-mon[100622]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:34:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:16 smithi184 ceph-mon[102844]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:34:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:16 smithi196 ceph-mon[76349]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:34:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:18 smithi196 ceph-mon[76349]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T21:34:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:18 smithi184 ceph-mon[100622]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T21:34:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:18 smithi184 ceph-mon[102844]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T21:34:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:20 smithi184 ceph-mon[100622]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:34:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:20 smithi184 ceph-mon[102844]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:34:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:20.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:20.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:20.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:20 smithi196 ceph-mon[76349]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:34:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:22 smithi184 ceph-mon[100622]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:34:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:22 smithi184 ceph-mon[102844]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:34:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:22 smithi196 ceph-mon[76349]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:34:24.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:23.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:23.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:23.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:25 smithi184 ceph-mon[100622]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T21:34:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:25 smithi184 ceph-mon[102844]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T21:34:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:25 smithi196 ceph-mon[76349]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T21:34:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:27 smithi184 ceph-mon[100622]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:27 smithi184 ceph-mon[102844]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:27 smithi196 ceph-mon[76349]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:29 smithi184 ceph-mon[100622]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:29 smithi184 ceph-mon[102844]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:29 smithi196 ceph-mon[76349]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T21:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:30.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:30.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:30.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:31 smithi184 ceph-mon[100622]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T21:34:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:31 smithi184 ceph-mon[102844]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T21:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:31 smithi196 ceph-mon[76349]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T21:34:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:34:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:33 smithi184 ceph-mon[100622]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:34:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:33 smithi184 ceph-mon[102844]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:34:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:33 smithi196 ceph-mon[76349]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:34:34.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:33.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:33.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:34.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:33.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:35.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:35 smithi184 ceph-mon[100622]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:34:35.136 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:35 smithi184 ceph-mon[102844]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:34:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:35 smithi196 ceph-mon[76349]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:34:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:37 smithi184 ceph-mon[100622]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T21:34:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:37 smithi184 ceph-mon[102844]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T21:34:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:37 smithi196 ceph-mon[76349]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T21:34:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:39 smithi184 ceph-mon[100622]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:34:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:39 smithi184 ceph-mon[102844]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:34:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:39 smithi196 ceph-mon[76349]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:34:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:40.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:40.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:40.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:40.863 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:34:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:41 smithi184 ceph-mon[100622]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:34:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:41 smithi184 ceph-mon[102844]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:41 smithi196 ceph-mon[76349]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:34:41.395 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:34:41.395 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 20s ago 111m - - 2024-09-20T21:34:41.395 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 111m - - 2024-09-20T21:34:41.395 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (105m) 20s ago 106m 30.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (106m) 18s ago 106m 92.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (99m) 20s ago 104m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (99m) 18s ago 110m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (100m) 20s ago 112m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (98m) 20s ago 112m 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (98m) 18s ago 110m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (98m) 20s ago 110m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (106m) 20s ago 106m 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (106m) 18s ago 106m 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (96m) 20s ago 110m 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (96m) 20s ago 109m 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (95m) 20s ago 109m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (94m) 20s ago 109m 79.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:34:41.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (93m) 18s ago 108m 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (92m) 18s ago 108m 66.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (91m) 18s ago 107m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (90m) 18s ago 107m 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (99m) 18s ago 107m 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (90m) 20s ago 105m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (90m) 18s ago 105m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (90m) 20s ago 104m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:34:41.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (90m) 18s ago 104m 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:34:41.714 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:34:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:34:41.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:34:41.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:34:41.983 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:34:41.984 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:34:42.298 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:34:42.298 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:42.298 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:42.298 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:42.298 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='client.60917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='client.60923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1079625518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='client.60917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='client.60923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1079625518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='client.60917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='client.60923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:42.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1079625518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:43 smithi196 ceph-mon[76349]: from='client.50961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:43 smithi196 ceph-mon[76349]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:34:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:43 smithi196 ceph-mon[76349]: from='client.50973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/820702856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[100622]: from='client.50961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[100622]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[100622]: from='client.50973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/820702856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[102844]: from='client.50961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[102844]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[102844]: from='client.50973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/820702856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:44.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:43.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:44.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:43.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:44.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:43.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:34:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:45 smithi196 ceph-mon[76349]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:45.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:45 smithi184 ceph-mon[100622]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:45.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:45 smithi184 ceph-mon[102844]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:47 smithi184 ceph-mon[100622]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:47 smithi184 ceph-mon[102844]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:47 smithi196 ceph-mon[76349]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:49 smithi184 ceph-mon[100622]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:49 smithi184 ceph-mon[102844]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:49 smithi196 ceph-mon[76349]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:50.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:50.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:50.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:51 smithi184 ceph-mon[100622]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:51 smithi184 ceph-mon[102844]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:51 smithi196 ceph-mon[76349]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:34:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:34:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:53 smithi184 ceph-mon[100622]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:53 smithi184 ceph-mon[102844]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:53 smithi196 ceph-mon[76349]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:54.216 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:53.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:54.216 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:53.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:34:54.216 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:34:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:34:53.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:34:55.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:55 smithi184 ceph-mon[100622]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:55.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:55 smithi184 ceph-mon[102844]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:55 smithi196 ceph-mon[76349]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:57 smithi184 ceph-mon[102844]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:57 smithi184 ceph-mon[100622]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:57 smithi196 ceph-mon[76349]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:59 smithi184 ceph-mon[102844]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:59 smithi184 ceph-mon[100622]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:59 smithi196 ceph-mon[76349]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:00.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:00.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:00.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:01 smithi184 ceph-mon[100622]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:01 smithi184 ceph-mon[102844]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:01 smithi196 ceph-mon[76349]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:03 smithi184 ceph-mon[100622]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:03 smithi184 ceph-mon[102844]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:03 smithi196 ceph-mon[76349]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:04.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:03.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:04.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:03.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:04.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:03.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[100622]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:04 smithi184 ceph-mon[102844]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:04 smithi196 ceph-mon[76349]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:06 smithi184 ceph-mon[100622]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:06 smithi184 ceph-mon[102844]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:06 smithi196 ceph-mon[76349]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:08 smithi184 ceph-mon[100622]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:08 smithi184 ceph-mon[102844]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:08.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:08 smithi196 ceph-mon[76349]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:10 smithi184 ceph-mon[100622]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:10 smithi184 ceph-mon[102844]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:10.523Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:10.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:10.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:10 smithi196 ceph-mon[76349]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:35:12.595 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:35:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:12 smithi184 ceph-mon[100622]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:12 smithi184 ceph-mon[102844]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:12 smithi196 ceph-mon[76349]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 111m - - 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 111m - - 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (106m) 11s ago 107m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (107m) 9s ago 107m 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:35:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (100m) 11s ago 105m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (99m) 9s ago 111m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (100m) 11s ago 112m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (99m) 11s ago 112m 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (98m) 9s ago 111m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (98m) 11s ago 111m 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (106m) 11s ago 106m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (106m) 9s ago 106m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (97m) 11s ago 110m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (96m) 11s ago 110m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (95m) 11s ago 109m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (95m) 11s ago 109m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (94m) 9s ago 109m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (93m) 9s ago 108m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (92m) 9s ago 108m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:35:13.126 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (91m) 9s ago 108m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:35:13.127 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (100m) 9s ago 107m 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:35:13.127 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (91m) 11s ago 105m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:35:13.127 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (90m) 9s ago 105m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:35:13.127 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (91m) 11s ago 105m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:35:13.127 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (90m) 9s ago 105m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:13.441 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:35:13.442 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:35:13.713 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:35:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:35:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:35:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:35:13.714 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:35:13.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:35:13.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[100622]: from='client.50985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[100622]: from='client.50991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[100622]: from='client.60965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3018058997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:13.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[102844]: from='client.50985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[102844]: from='client.50991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[102844]: from='client.60965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3018058997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:13 smithi196 ceph-mon[76349]: from='client.50985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:13 smithi196 ceph-mon[76349]: from='client.50991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:13 smithi196 ceph-mon[76349]: from='client.60965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3018058997' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:14.024 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:35:14.024 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:14.025 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:14.026 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:13.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:14.026 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:13.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:14.026 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:13.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[100622]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[100622]: from='client.60977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3792481493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[102844]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[102844]: from='client.60977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3792481493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:14 smithi196 ceph-mon[76349]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:14 smithi196 ceph-mon[76349]: from='client.60977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3792481493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:16 smithi184 ceph-mon[100622]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:16 smithi184 ceph-mon[102844]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:16 smithi196 ceph-mon[76349]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:18 smithi184 ceph-mon[100622]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:18 smithi184 ceph-mon[102844]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:18.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:18 smithi196 ceph-mon[76349]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:20 smithi184 ceph-mon[100622]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:20 smithi184 ceph-mon[102844]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:20.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:20.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:20.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:20 smithi196 ceph-mon[76349]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:35:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[102844]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[100622]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:22 smithi196 ceph-mon[76349]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:23.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:23.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:23.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:24 smithi196 ceph-mon[76349]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[100622]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:24 smithi184 ceph-mon[102844]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:26.708 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:26 smithi196 ceph-mon[76349]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:26 smithi184 ceph-mon[100622]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:26 smithi184 ceph-mon[102844]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:28 smithi184 ceph-mon[100622]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:28 smithi184 ceph-mon[102844]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:28 smithi196 ceph-mon[76349]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:30 smithi184 ceph-mon[100622]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:30 smithi184 ceph-mon[102844]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:30.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:30.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:30.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:30 smithi196 ceph-mon[76349]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:35:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:32 smithi184 ceph-mon[100622]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:32 smithi184 ceph-mon[102844]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:32 smithi196 ceph-mon[76349]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:33.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:33.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:33.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:34 smithi184 ceph-mon[100622]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:34 smithi184 ceph-mon[102844]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:34 smithi196 ceph-mon[76349]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:36.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:36 smithi184 ceph-mon[100622]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:36.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:36 smithi184 ceph-mon[102844]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:36 smithi196 ceph-mon[76349]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:38 smithi184 ceph-mon[100622]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:38 smithi184 ceph-mon[102844]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:38 smithi196 ceph-mon[76349]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:40 smithi184 ceph-mon[100622]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:40 smithi184 ceph-mon[102844]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:40.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:40.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:40.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:40 smithi196 ceph-mon[76349]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:35:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:42 smithi196 ceph-mon[76349]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[100622]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[102844]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:43.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:43.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:43.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:44.320 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:35:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[102844]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[100622]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 112m - - 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 112m - - 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (106m) 2s ago 107m 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (107m) 1s ago 107m 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (100m) 2s ago 105m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (100m) 1s ago 111m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (101m) 2s ago 113m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:35:44.850 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (99m) 2s ago 113m 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (99m) 1s ago 112m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (99m) 2s ago 111m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (107m) 2s ago 107m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (107m) 1s ago 107m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (98m) 2s ago 111m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (97m) 2s ago 110m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (96m) 2s ago 110m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (95m) 2s ago 110m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (94m) 1s ago 109m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (93m) 1s ago 109m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (92m) 1s ago 108m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (92m) 1s ago 108m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:35:44.851 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (100m) 1s ago 108m 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:35:44.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (91m) 2s ago 106m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:35:44.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (91m) 1s ago 106m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:35:44.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (91m) 2s ago 105m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:35:44.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (91m) 1s ago 105m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:44 smithi196 ceph-mon[76349]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:35:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:45.158 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:35:45.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:35:45.159 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:35:45.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:35:45.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:35:45.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:35:45.427 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:35:45.428 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:35:45.429 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:35:45.429 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:35:45.429 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:35:45.740 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:35:45.740 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:45.740 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:45.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[100622]: from='client.51021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[100622]: from='client.60995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[100622]: from='client.61001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3731412946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:45.741 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[102844]: from='client.51021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.741 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[102844]: from='client.60995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.741 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[102844]: from='client.61001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.741 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3731412946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:45 smithi196 ceph-mon[76349]: from='client.51021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:45 smithi196 ceph-mon[76349]: from='client.60995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:45 smithi196 ceph-mon[76349]: from='client.61001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3731412946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[100622]: from='client.51045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[100622]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/730602176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[102844]: from='client.51045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[102844]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/730602176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:46 smithi196 ceph-mon[76349]: from='client.51045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:46 smithi196 ceph-mon[76349]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/730602176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:48 smithi184 ceph-mon[100622]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:48 smithi184 ceph-mon[102844]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:48.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:48 smithi196 ceph-mon[76349]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:50 smithi184 ceph-mon[100622]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:50 smithi184 ceph-mon[102844]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:50.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:50.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:50.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:50 smithi196 ceph-mon[76349]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:35:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:35:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:52 smithi184 ceph-mon[100622]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:52 smithi184 ceph-mon[102844]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:52 smithi196 ceph-mon[76349]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:53.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:53.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:35:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:35:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:35:53.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:35:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:54 smithi184 ceph-mon[100622]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:54 smithi184 ceph-mon[102844]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:54 smithi196 ceph-mon[76349]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:56 smithi184 ceph-mon[102844]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:56 smithi184 ceph-mon[100622]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:56 smithi196 ceph-mon[76349]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:58.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:58 smithi184 ceph-mon[100622]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:58.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:58 smithi184 ceph-mon[102844]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:58.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:58 smithi196 ceph-mon[76349]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:00 smithi184 ceph-mon[100622]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:00 smithi184 ceph-mon[102844]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:00.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:00.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:00.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:00 smithi196 ceph-mon[76349]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:01.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:01.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[100622]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[102844]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:02 smithi196 ceph-mon[76349]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:03.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:03.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:03.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[100622]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[102844]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:04 smithi196 ceph-mon[76349]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:05.151 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.151 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:05.151 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:06.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:06 smithi196 ceph-mon[76349]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:06 smithi184 ceph-mon[100622]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:06 smithi184 ceph-mon[102844]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:08 smithi184 ceph-mon[100622]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:08 smithi184 ceph-mon[102844]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:08 smithi196 ceph-mon[76349]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:10 smithi184 ceph-mon[100622]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:10 smithi184 ceph-mon[102844]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:10.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:10.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:10.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:11.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:10 smithi196 ceph-mon[76349]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:36:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:12 smithi184 ceph-mon[100622]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:12 smithi184 ceph-mon[102844]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:12 smithi196 ceph-mon[76349]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:13.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:13.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:13.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:14 smithi184 ceph-mon[102844]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:14 smithi184 ceph-mon[100622]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:14 smithi196 ceph-mon[76349]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:16.038 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 113m - - 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 112m - - 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (107m) 14s ago 108m 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (108m) 12s ago 108m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:36:16.561 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (101m) 14s ago 106m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (101m) 12s ago 112m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (101m) 14s ago 113m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (100m) 14s ago 113m 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (99m) 12s ago 112m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (99m) 14s ago 112m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (107m) 14s ago 107m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (107m) 12s ago 107m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (98m) 14s ago 111m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (97m) 14s ago 111m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (97m) 14s ago 111m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (96m) 14s ago 110m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (95m) 12s ago 110m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (94m) 12s ago 109m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (93m) 12s ago 109m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:36:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (92m) 12s ago 109m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:36:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (101m) 12s ago 108m 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:36:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (92m) 14s ago 106m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:36:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (92m) 12s ago 106m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:36:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (92m) 14s ago 106m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:36:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (91m) 12s ago 106m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:36:16.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:16 smithi184 ceph-mon[100622]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:16.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:16 smithi184 ceph-mon[100622]: from='client.61025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:16.831 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:16 smithi184 ceph-mon[102844]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:16.831 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:16 smithi184 ceph-mon[102844]: from='client.61025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:16.878 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:36:16.878 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:36:16.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:36:16.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:36:16.879 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:36:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:16 smithi196 ceph-mon[76349]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:16 smithi196 ceph-mon[76349]: from='client.61025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:36:17.473 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:36:17.473 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:17.473 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:17.473 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:17.473 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[102844]: from='client.51063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[102844]: from='client.61037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2998608940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[102844]: from='client.61049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/19756075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[100622]: from='client.51063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[100622]: from='client.61037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2998608940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[100622]: from='client.61049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/19756075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:17 smithi196 ceph-mon[76349]: from='client.51063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:17 smithi196 ceph-mon[76349]: from='client.61037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2998608940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:17 smithi196 ceph-mon[76349]: from='client.61049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/19756075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:18 smithi184 ceph-mon[100622]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:18 smithi184 ceph-mon[102844]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:18 smithi196 ceph-mon[76349]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:20.678 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:20.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:20.678 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:20.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:20.678 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:20.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:20 smithi184 ceph-mon[100622]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:20 smithi184 ceph-mon[102844]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:20 smithi196 ceph-mon[76349]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:36:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:23 smithi196 ceph-mon[76349]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[100622]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[102844]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:23.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:23.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:23.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[100622]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[102844]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:24 smithi196 ceph-mon[76349]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:25.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:26 smithi184 ceph-mon[100622]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:26 smithi184 ceph-mon[102844]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:26 smithi196 ceph-mon[76349]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:28 smithi184 ceph-mon[100622]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:28 smithi184 ceph-mon[102844]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:28 smithi196 ceph-mon[76349]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:29.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:30.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:30.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:30.788 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:30.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:30.788 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:30.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:30 smithi184 ceph-mon[100622]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:30 smithi184 ceph-mon[102844]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:30 smithi196 ceph-mon[76349]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:32.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:36:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:32 smithi184 ceph-mon[100622]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:32 smithi184 ceph-mon[102844]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:32 smithi196 ceph-mon[76349]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:34.226 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:33.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:34.227 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:33.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:34.227 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:33.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:34 smithi196 ceph-mon[76349]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:36:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:34 smithi184 ceph-mon[100622]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:36:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:34 smithi184 ceph-mon[102844]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:36:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:36 smithi196 ceph-mon[76349]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:36 smithi184 ceph-mon[100622]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:36 smithi184 ceph-mon[102844]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:38.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:38 smithi184 ceph-mon[100622]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:38.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:38 smithi184 ceph-mon[102844]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:38 smithi196 ceph-mon[76349]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:40.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:40.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:40.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:40 smithi196 ceph-mon[76349]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:40 smithi184 ceph-mon[100622]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:40 smithi184 ceph-mon[102844]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:42.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:42.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:42.268 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:36:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[100622]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[102844]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:43 smithi196 ceph-mon[76349]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:43.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:44.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:43.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:44.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:43.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:36:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:45 smithi184 ceph-mon[100622]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:45 smithi184 ceph-mon[102844]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:45 smithi196 ceph-mon[76349]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:47 smithi184 ceph-mon[100622]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:47 smithi184 ceph-mon[102844]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:47 smithi196 ceph-mon[76349]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:47.768 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 113m - - 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 113m - - 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (107m) 6s ago 108m 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (108m) 4s ago 108m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (101m) 6s ago 106m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (101m) 4s ago 112m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:36:48.298 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (102m) 6s ago 114m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (100m) 6s ago 114m 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (100m) 4s ago 113m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (100m) 6s ago 112m 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (108m) 6s ago 108m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (108m) 4s ago 108m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (99m) 6s ago 112m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (98m) 6s ago 111m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (97m) 6s ago 111m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (97m) 6s ago 111m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:36:48.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (95m) 4s ago 110m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (94m) 4s ago 110m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (93m) 4s ago 110m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (93m) 4s ago 109m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (101m) 4s ago 109m 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (92m) 6s ago 107m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (92m) 4s ago 107m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (92m) 6s ago 106m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:36:48.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (92m) 4s ago 106m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:36:48.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:36:48.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:36:48.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:36:48.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:36:48.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:36:48.876 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:36:49.183 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:36:49.184 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:49.184 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:49.184 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:49.184 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[100622]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[100622]: from='client.51093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[100622]: from='client.61067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2502650467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3943751133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[102844]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[102844]: from='client.51093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[102844]: from='client.61067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2502650467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3943751133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:49 smithi196 ceph-mon[76349]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:49 smithi196 ceph-mon[76349]: from='client.51093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:49 smithi196 ceph-mon[76349]: from='client.61067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2502650467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3943751133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:50 smithi196 ceph-mon[76349]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:50 smithi196 ceph-mon[76349]: from='client.51117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-mon[100622]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-mon[100622]: from='client.51117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-mon[102844]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-mon[102844]: from='client.51117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:50.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:50.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:50.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:51 smithi184 ceph-mon[100622]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:51 smithi184 ceph-mon[102844]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:51 smithi196 ceph-mon[76349]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:52.274 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:36:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:36:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:52 smithi184 ceph-mon[100622]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:52 smithi184 ceph-mon[102844]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:52 smithi196 ceph-mon[76349]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:53.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:53.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:36:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:36:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:36:53.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:36:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:54 smithi184 ceph-mon[100622]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:54 smithi184 ceph-mon[102844]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:54 smithi196 ceph-mon[76349]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:56 smithi184 ceph-mon[102844]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:56 smithi184 ceph-mon[100622]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:56 smithi196 ceph-mon[76349]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:58 smithi184 ceph-mon[100622]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:58 smithi184 ceph-mon[102844]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:58 smithi196 ceph-mon[76349]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:59.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:59.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:00 smithi184 ceph-mon[100622]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:00 smithi184 ceph-mon[102844]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:00.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:00.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:00.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:00 smithi196 ceph-mon[76349]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:37:02.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[100622]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:02 smithi196 ceph-mon[76349]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[102844]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:03.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:03.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:03.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[100622]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[102844]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:05 smithi196 ceph-mon[76349]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:07 smithi184 ceph-mon[100622]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:07 smithi184 ceph-mon[102844]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:07 smithi196 ceph-mon[76349]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:09 smithi196 ceph-mon[76349]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:09 smithi184 ceph-mon[100622]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:09 smithi184 ceph-mon[102844]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:10.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:10.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:10.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:11 smithi184 ceph-mon[100622]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:11 smithi184 ceph-mon[102844]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:11 smithi196 ceph-mon[76349]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:37:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:13 smithi184 ceph-mon[100622]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:13 smithi184 ceph-mon[102844]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:13 smithi196 ceph-mon[76349]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:13.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:13.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:13.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:15 smithi196 ceph-mon[76349]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:15 smithi184 ceph-mon[100622]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:15 smithi184 ceph-mon[102844]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:17 smithi196 ceph-mon[76349]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:17 smithi184 ceph-mon[102844]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:17 smithi184 ceph-mon[100622]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:19.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:19 smithi184 ceph-mon[100622]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:19.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:19 smithi184 ceph-mon[102844]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:19 smithi196 ceph-mon[76349]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:19.484 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 114m - - 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 113m - - 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (108m) 17s ago 109m 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (109m) 15s ago 109m 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (102m) 17s ago 107m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (102m) 15s ago 113m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (102m) 17s ago 114m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:37:20.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (101m) 17s ago 115m 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (100m) 15s ago 113m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (101m) 17s ago 113m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (108m) 17s ago 108m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (108m) 15s ago 108m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (99m) 17s ago 112m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (98m) 17s ago 112m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (98m) 17s ago 112m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (97m) 17s ago 111m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (96m) 15s ago 111m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (95m) 15s ago 110m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (94m) 15s ago 110m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (93m) 15s ago 110m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (102m) 15s ago 109m 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:37:20.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (93m) 17s ago 107m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:37:20.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (93m) 15s ago 107m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:37:20.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (93m) 17s ago 107m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:37:20.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (92m) 15s ago 107m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:37:20.351 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:37:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:37:20.351 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:37:20.352 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:37:20.616 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:20.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:20.617 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:20.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:20.617 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:20.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:37:20.630 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:37:20.631 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:37:20.987 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:37:20.987 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:20.987 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:20.987 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:20.987 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:21.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: from='client.51129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:21.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: from='client.61109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3936505242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3855426288' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: from='client.51129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: from='client.61109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3936505242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:21.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3855426288' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: from='client.51129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: from='client.61109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3936505242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3855426288' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:22 smithi184 ceph-mon[100622]: from='client.51153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:37:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:22 smithi184 ceph-mon[102844]: from='client.51153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:22 smithi196 ceph-mon[76349]: from='client.51153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:23 smithi196 ceph-mon[76349]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[100622]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[102844]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:23.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:23.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:23.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[100622]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[102844]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:25 smithi196 ceph-mon[76349]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:27 smithi184 ceph-mon[100622]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:27 smithi184 ceph-mon[102844]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:27 smithi196 ceph-mon[76349]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:29.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:29 smithi196 ceph-mon[76349]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:29.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:29 smithi184 ceph-mon[100622]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:29 smithi184 ceph-mon[102844]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:30.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:30.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:30.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:31 smithi196 ceph-mon[76349]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:31 smithi184 ceph-mon[100622]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:31 smithi184 ceph-mon[102844]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:37:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:33 smithi184 ceph-mon[102844]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:33 smithi184 ceph-mon[100622]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:33 smithi196 ceph-mon[76349]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:33.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:33.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:33.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:35 smithi184 ceph-mon[102844]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:35 smithi184 ceph-mon[100622]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:35 smithi196 ceph-mon[76349]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:37 smithi184 ceph-mon[102844]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:37 smithi184 ceph-mon[100622]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:37 smithi196 ceph-mon[76349]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:39.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:39 smithi196 ceph-mon[76349]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:39 smithi184 ceph-mon[102844]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:39 smithi184 ceph-mon[100622]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:40.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:40.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:40.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:40.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:40.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:40.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:41.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:41 smithi184 ceph-mon[100622]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:41.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:41 smithi184 ceph-mon[102844]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:41 smithi196 ceph-mon[76349]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:42.176 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:37:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:43.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[100622]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[102844]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:43 smithi196 ceph-mon[76349]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:43.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:43.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:43.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:43.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[100622]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[102844]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:45 smithi196 ceph-mon[76349]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:37:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:47 smithi184 ceph-mon[100622]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:47 smithi184 ceph-mon[102844]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:47 smithi196 ceph-mon[76349]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:49 smithi184 ceph-mon[100622]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:49 smithi184 ceph-mon[102844]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:49 smithi196 ceph-mon[76349]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:50.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:50.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:50.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:51.286 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:37:51.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:51 smithi196 ceph-mon[76349]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:51 smithi184 ceph-mon[100622]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:51 smithi184 ceph-mon[102844]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:51.812 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:37:51.812 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 114m - - 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 114m - - 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (108m) 9s ago 109m 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (109m) 7s ago 109m 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (102m) 9s ago 107m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (102m) 7s ago 113m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (103m) 9s ago 115m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (101m) 9s ago 115m 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (101m) 7s ago 114m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (101m) 9s ago 114m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (109m) 9s ago 109m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (109m) 7s ago 109m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:37:51.813 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (100m) 9s ago 113m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (99m) 9s ago 113m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (98m) 9s ago 112m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (98m) 9s ago 112m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (96m) 7s ago 111m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (95m) 7s ago 111m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (94m) 7s ago 111m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (94m) 7s ago 110m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (102m) 7s ago 110m 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (93m) 9s ago 108m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (93m) 7s ago 108m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (93m) 9s ago 107m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:37:51.814 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (93m) 7s ago 107m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:37:52.126 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:37:52.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:37:52.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:37:52.128 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:37:52.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:37:52.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:37:52.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:37:52.128 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:37:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:37:52.403 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:37:52.403 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:37:52.403 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:37:52.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[100622]: from='client.61133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4216702588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[102844]: from='client.61133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:52.405 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4216702588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:52.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:52.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:52 smithi196 ceph-mon[76349]: from='client.61133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:52.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4216702588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:52.715 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:37:52.715 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:52.716 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:52.716 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:52.716 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[100622]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[100622]: from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[100622]: from='client.51177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2377677250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[102844]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[102844]: from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[102844]: from='client.51177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2377677250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:53 smithi196 ceph-mon[76349]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:53 smithi196 ceph-mon[76349]: from='client.61139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:53 smithi196 ceph-mon[76349]: from='client.51177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2377677250' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:54.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:53.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:54.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:53.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:37:54.223 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:37:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:37:53.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:37:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:54 smithi196 ceph-mon[76349]: from='client.61157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:54 smithi184 ceph-mon[102844]: from='client.61157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:54.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:54 smithi184 ceph-mon[100622]: from='client.61157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:55 smithi184 ceph-mon[100622]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:55 smithi184 ceph-mon[102844]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:55 smithi196 ceph-mon[76349]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:57 smithi184 ceph-mon[100622]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:57 smithi184 ceph-mon[102844]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:57 smithi196 ceph-mon[76349]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:59 smithi184 ceph-mon[102844]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:59 smithi184 ceph-mon[100622]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:59 smithi196 ceph-mon[76349]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:00.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:00.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:00.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:00.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:00.629 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:00.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:01.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:01 smithi184 ceph-mon[100622]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:01.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:01 smithi184 ceph-mon[102844]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:01 smithi196 ceph-mon[76349]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:02.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:38:02.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:02.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[100622]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[102844]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:03 smithi196 ceph-mon[76349]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:03.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:03.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:03.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:03.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[100622]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[102844]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:05 smithi196 ceph-mon[76349]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:05.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:07 smithi184 ceph-mon[100622]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:07 smithi184 ceph-mon[102844]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:07 smithi196 ceph-mon[76349]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:09 smithi184 ceph-mon[100622]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:09 smithi184 ceph-mon[102844]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:09 smithi196 ceph-mon[76349]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:10.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:10.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:10.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:11.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:11 smithi196 ceph-mon[76349]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:11 smithi184 ceph-mon[100622]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:11 smithi184 ceph-mon[102844]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:12.258 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:38:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:13 smithi196 ceph-mon[76349]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:13 smithi184 ceph-mon[100622]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:13 smithi184 ceph-mon[102844]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:38:13.812733199Z level=info msg="Completed cleanup jobs" duration=12.857304ms 2024-09-20T21:38:14.265 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:13.971Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:14.265 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:13.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:14.265 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:13.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:15 smithi184 ceph-mon[100622]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:15 smithi184 ceph-mon[102844]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:15 smithi196 ceph-mon[76349]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:17 smithi184 ceph-mon[100622]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:17 smithi184 ceph-mon[102844]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:17 smithi196 ceph-mon[76349]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:19 smithi184 ceph-mon[100622]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:19 smithi184 ceph-mon[102844]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:19 smithi196 ceph-mon[76349]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:20.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:20.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:20.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:21 smithi196 ceph-mon[76349]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:21 smithi184 ceph-mon[100622]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:21 smithi184 ceph-mon[102844]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:22.243 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:38:22.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:23.031 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:38:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[100622]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[102844]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 115m - - 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 114m - - 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (109m) 0s ago 110m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (110m) 19s ago 110m 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (103m) 0s ago 108m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (103m) 19s ago 114m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (104m) 0s ago 115m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (102m) 0s ago 116m 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (101m) 19s ago 114m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (102m) 0s ago 114m 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:38:23.570 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (109m) 0s ago 109m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (109m) 19s ago 109m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (100m) 0s ago 113m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (99m) 0s ago 113m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (99m) 0s ago 113m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (98m) 0s ago 112m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (97m) 19s ago 112m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (96m) 19s ago 112m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (95m) 19s ago 111m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (94m) 19s ago 111m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (103m) 19s ago 110m 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (94m) 0s ago 108m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (94m) 19s ago 108m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:38:23.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (94m) 0s ago 108m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:38:23.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (94m) 19s ago 108m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:38:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:23 smithi196 ceph-mon[76349]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:23.883 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:38:23.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:38:24.150 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:38:24.151 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:38:24.151 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:38:24.151 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:38:24.151 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:38:24.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:38:24.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:23.971Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:24.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:23.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:24.151 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:23.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:24.464 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:38:24.464 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:24.464 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:24.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:24.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: from='client.61169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: from='client.51207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/650563935' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[102844]: from='client.51225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: from='client.61169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: from='client.51207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/650563935' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:24.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:24 smithi184 ceph-mon[100622]: from='client.51225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: from='client.61169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: from='client.51207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/650563935' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:24.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:24 smithi196 ceph-mon[76349]: from='client.51225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/330055002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/330055002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/330055002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:26 smithi196 ceph-mon[76349]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:26 smithi184 ceph-mon[100622]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:26.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:26 smithi184 ceph-mon[102844]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:28 smithi184 ceph-mon[100622]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:28 smithi184 ceph-mon[102844]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:28 smithi196 ceph-mon[76349]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:30 smithi184 ceph-mon[100622]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:30 smithi184 ceph-mon[102844]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:30.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:30.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:30.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:30 smithi196 ceph-mon[76349]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:38:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:32 smithi184 ceph-mon[100622]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:32 smithi184 ceph-mon[102844]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:32 smithi196 ceph-mon[76349]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:33.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:33.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:33.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:34 smithi184 ceph-mon[100622]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:34 smithi184 ceph-mon[102844]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:34 smithi196 ceph-mon[76349]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:36 smithi184 ceph-mon[100622]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:36 smithi184 ceph-mon[102844]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:36 smithi196 ceph-mon[76349]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:38 smithi184 ceph-mon[100622]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:38 smithi184 ceph-mon[102844]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:38 smithi196 ceph-mon[76349]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:40 smithi184 ceph-mon[102844]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:40 smithi184 ceph-mon[100622]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:40.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:40.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:40.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:40.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:40 smithi196 ceph-mon[76349]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:38:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:42 smithi184 ceph-mon[100622]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:42 smithi184 ceph-mon[102844]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:42 smithi196 ceph-mon[76349]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:44.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:43.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:44.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:43.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:44.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:43.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[102844]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[100622]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:45 smithi196 ceph-mon[76349]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:45.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:38:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:47 smithi184 ceph-mon[100622]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:47 smithi184 ceph-mon[102844]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:47 smithi196 ceph-mon[76349]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:49 smithi184 ceph-mon[102844]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:49 smithi184 ceph-mon[100622]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:49 smithi196 ceph-mon[76349]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:50.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:50.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:50.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:51 smithi184 ceph-mon[100622]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:51 smithi184 ceph-mon[102844]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:51 smithi196 ceph-mon[76349]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:38:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:38:53.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:53 smithi196 ceph-mon[76349]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:53 smithi184 ceph-mon[100622]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:53 smithi184 ceph-mon[102844]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:53.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:53.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:38:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:38:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:38:53.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:38:54.761 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 115m - - 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 115m - - 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (109m) 12s ago 110m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (110m) 10s ago 110m 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (103m) 12s ago 108m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (103m) 10s ago 114m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (104m) 12s ago 116m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (102m) 12s ago 116m 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (102m) 10s ago 115m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:38:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (102m) 12s ago 115m 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (110m) 12s ago 110m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (110m) 10s ago 110m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (101m) 12s ago 114m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (100m) 12s ago 114m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (99m) 12s ago 113m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (99m) 12s ago 113m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (97m) 10s ago 112m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (97m) 10s ago 112m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (95m) 10s ago 112m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (95m) 10s ago 111m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (104m) 10s ago 111m 72.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (94m) 12s ago 109m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (94m) 10s ago 109m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:38:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (94m) 12s ago 109m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:38:55.298 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (94m) 10s ago 108m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:38:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:55 smithi184 ceph-mon[102844]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:55 smithi184 ceph-mon[100622]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:55 smithi196 ceph-mon[76349]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:55.606 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:38:55.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:38:55.608 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:38:55.608 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:38:55.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:38:55.608 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:38:55.878 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:38:55.879 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:38:55.879 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:38:55.879 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:38:55.879 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:38:56.187 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:38:56.188 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:56.188 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:56.188 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:56.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[100622]: from='client.51237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[100622]: from='client.61211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[100622]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2614573200' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:56.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[102844]: from='client.51237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[102844]: from='client.61211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[102844]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2614573200' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:56 smithi196 ceph-mon[76349]: from='client.51237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:56 smithi196 ceph-mon[76349]: from='client.61211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:56 smithi196 ceph-mon[76349]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2614573200' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:57 smithi196 ceph-mon[76349]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:57 smithi196 ceph-mon[76349]: from='client.61229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1992854255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[100622]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[100622]: from='client.61229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1992854255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[102844]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[102844]: from='client.61229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1992854255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:59 smithi196 ceph-mon[76349]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:59 smithi184 ceph-mon[100622]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:59 smithi184 ceph-mon[102844]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:00.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:00.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:00.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:01 smithi196 ceph-mon[76349]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:01 smithi184 ceph-mon[100622]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:01 smithi184 ceph-mon[102844]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:39:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:03 smithi196 ceph-mon[76349]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:03.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:03 smithi184 ceph-mon[100622]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:03.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:03 smithi184 ceph-mon[102844]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:03.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:03.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:04.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:03.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:04.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[100622]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[102844]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:05 smithi196 ceph-mon[76349]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:07 smithi184 ceph-mon[100622]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:07 smithi184 ceph-mon[102844]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:07 smithi196 ceph-mon[76349]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:09 smithi184 ceph-mon[100622]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:09 smithi184 ceph-mon[102844]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:09 smithi196 ceph-mon[76349]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:10.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:10.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:10.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:11.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:11 smithi196 ceph-mon[76349]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:11 smithi184 ceph-mon[102844]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:11 smithi184 ceph-mon[100622]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:39:12.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:13.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:13 smithi196 ceph-mon[76349]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:13 smithi184 ceph-mon[100622]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:13 smithi184 ceph-mon[102844]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:13.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:13.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:13.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:14.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:15 smithi184 ceph-mon[100622]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:15 smithi184 ceph-mon[102844]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:15 smithi196 ceph-mon[76349]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:17 smithi184 ceph-mon[100622]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:17 smithi184 ceph-mon[102844]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:17 smithi196 ceph-mon[76349]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:19 smithi184 ceph-mon[100622]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:19 smithi184 ceph-mon[102844]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:19 smithi196 ceph-mon[76349]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:20.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:20.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:20.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:21 smithi184 ceph-mon[100622]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:21 smithi184 ceph-mon[102844]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:21 smithi196 ceph-mon[76349]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:39:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:23.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:23 smithi184 ceph-mon[100622]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:23 smithi184 ceph-mon[102844]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:23 smithi196 ceph-mon[76349]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:24.157 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:23.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:24.157 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:23.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:24.157 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:23.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[102844]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[100622]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:25 smithi196 ceph-mon[76349]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:26.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:26 smithi184 ceph-mon[100622]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:26.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:26 smithi184 ceph-mon[102844]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:26.486 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:39:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:26 smithi196 ceph-mon[76349]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:27.023 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 116m - - 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 115m - - 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (110m) 3s ago 111m 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (111m) 2s ago 111m 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (104m) 3s ago 109m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (104m) 2s ago 115m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (105m) 3s ago 117m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (103m) 3s ago 117m 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (102m) 2s ago 115m 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (103m) 3s ago 115m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (111m) 3s ago 110m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:39:27.024 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (110m) 2s ago 110m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (101m) 3s ago 114m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (100m) 3s ago 114m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (100m) 3s ago 114m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (99m) 3s ago 113m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (98m) 2s ago 113m 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (97m) 2s ago 113m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (96m) 2s ago 112m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (95m) 2s ago 112m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (104m) 2s ago 111m 72.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (95m) 3s ago 109m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (95m) 2s ago 110m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (95m) 3s ago 109m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:39:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (95m) 2s ago 109m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:39:27.337 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:39:27.337 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:39:27.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:39:27.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:39:27.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:39:27.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[102844]: from='client.51273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[102844]: from='client.61247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[102844]: from='client.61253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:39:27.609 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:39:27.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[100622]: from='client.51273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[100622]: from='client.61247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.610 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:27 smithi184 ceph-mon[100622]: from='client.61253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:27 smithi196 ceph-mon[76349]: from='client.51273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:27 smithi196 ceph-mon[76349]: from='client.61247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:27 smithi196 ceph-mon[76349]: from='client.61253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:27.920 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:39:27.921 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:27.921 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/948171766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:28 smithi196 ceph-mon[76349]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:28 smithi196 ceph-mon[76349]: from='client.61265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/817051238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/948171766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[100622]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[100622]: from='client.61265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/817051238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/948171766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[102844]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[102844]: from='client.61265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/817051238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:30.576 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:30 smithi184 ceph-mon[100622]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:30.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:30 smithi184 ceph-mon[102844]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:30.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:30.543Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:30.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:30.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:30.577 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:30.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:30 smithi196 ceph-mon[76349]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:39:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:32 smithi196 ceph-mon[76349]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:32 smithi184 ceph-mon[100622]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:32 smithi184 ceph-mon[102844]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:33.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:33.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:33.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:34 smithi196 ceph-mon[76349]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:34 smithi184 ceph-mon[100622]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:34 smithi184 ceph-mon[102844]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:36 smithi196 ceph-mon[76349]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:36 smithi184 ceph-mon[100622]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:36 smithi184 ceph-mon[102844]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:38 smithi196 ceph-mon[76349]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:38 smithi184 ceph-mon[100622]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:38 smithi184 ceph-mon[102844]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:40 smithi184 ceph-mon[100622]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:40 smithi184 ceph-mon[102844]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:40.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:40.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:40.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:40.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:40 smithi196 ceph-mon[76349]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:42.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:42.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:39:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:42 smithi184 ceph-mon[100622]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:42 smithi184 ceph-mon[102844]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:42 smithi196 ceph-mon[76349]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:43.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:43.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:43.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:44.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[100622]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[102844]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:44.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:44 smithi196 ceph-mon[76349]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:44.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:39:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:47 smithi196 ceph-mon[76349]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:47 smithi184 ceph-mon[100622]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:47 smithi184 ceph-mon[102844]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:49 smithi196 ceph-mon[76349]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:49 smithi184 ceph-mon[100622]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:49 smithi184 ceph-mon[102844]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:50.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:50.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:50.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:51 smithi196 ceph-mon[76349]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:51 smithi184 ceph-mon[100622]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:51 smithi184 ceph-mon[102844]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:39:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:39:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:53 smithi196 ceph-mon[76349]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:53 smithi184 ceph-mon[100622]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:53 smithi184 ceph-mon[102844]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:53.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:53.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:39:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:39:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:39:53.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:39:55.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:55 smithi196 ceph-mon[76349]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:55 smithi184 ceph-mon[100622]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:55 smithi184 ceph-mon[102844]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:57 smithi196 ceph-mon[76349]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:57 smithi184 ceph-mon[100622]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:57 smithi184 ceph-mon[102844]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:58.215 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:39:58.741 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:39:58.741 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 116m - - 2024-09-20T21:39:58.741 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 116m - - 2024-09-20T21:39:58.741 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (110m) 15s ago 111m 29.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:39:58.741 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (111m) 13s ago 111m 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (104m) 15s ago 109m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (104m) 13s ago 115m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (105m) 15s ago 117m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (103m) 15s ago 117m 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (103m) 13s ago 116m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (103m) 15s ago 116m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (111m) 15s ago 111m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (111m) 13s ago 111m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (102m) 15s ago 115m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (101m) 15s ago 115m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (100m) 15s ago 114m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (100m) 15s ago 114m 79.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (98m) 13s ago 113m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:39:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (98m) 13s ago 113m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (96m) 13s ago 113m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (96m) 13s ago 112m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (105m) 13s ago 112m 72.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (95m) 15s ago 110m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (95m) 13s ago 110m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (95m) 15s ago 110m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:39:58.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (95m) 13s ago 110m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:39:59.056 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:39:59.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:39:59.330 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:39:59.330 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:39:59.330 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:39:59.330 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:39:59.330 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:39:59.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[100622]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:59.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[100622]: from='client.61277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:59.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1007713792' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:59.331 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[102844]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:59.331 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[102844]: from='client.61277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:59.331 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.331 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:39:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1007713792' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:59 smithi196 ceph-mon[76349]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:59 smithi196 ceph-mon[76349]: from='client.61277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:39:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1007713792' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:59.640 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:39:59.640 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:59.640 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[102844]: from='client.51315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[102844]: from='client.51321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3659929229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:39:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[100622]: from='client.51315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[100622]: from='client.51321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3659929229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:00.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:00 smithi196 ceph-mon[76349]: from='client.51315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:00 smithi196 ceph-mon[76349]: from='client.51321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3659929229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:00.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:00.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:00.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:01 smithi196 ceph-mon[76349]: from='client.61301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:01 smithi196 ceph-mon[76349]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:01 smithi184 ceph-mon[100622]: from='client.61301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:01 smithi184 ceph-mon[100622]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:01 smithi184 ceph-mon[102844]: from='client.61301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:01 smithi184 ceph-mon[102844]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:40:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:03.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:03 smithi196 ceph-mon[76349]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:03.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:03 smithi184 ceph-mon[100622]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:03.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:03 smithi184 ceph-mon[102844]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.280 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:03.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:04.280 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:03.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:04.280 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:03.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[100622]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[102844]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:04 smithi196 ceph-mon[76349]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:07 smithi184 ceph-mon[100622]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:07 smithi184 ceph-mon[102844]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:07 smithi196 ceph-mon[76349]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:09 smithi184 ceph-mon[100622]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:09 smithi184 ceph-mon[102844]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:09 smithi196 ceph-mon[76349]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:10.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:10.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:10.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:11.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:11 smithi196 ceph-mon[76349]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:11 smithi184 ceph-mon[100622]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:11 smithi184 ceph-mon[102844]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:40:12.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:13 smithi184 ceph-mon[100622]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:13 smithi184 ceph-mon[102844]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:13 smithi196 ceph-mon[76349]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:13.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:13.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:13.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:14.506 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:15.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:15 smithi196 ceph-mon[76349]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:15 smithi184 ceph-mon[100622]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:15 smithi184 ceph-mon[102844]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:17 smithi184 ceph-mon[100622]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:17 smithi184 ceph-mon[102844]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:17 smithi196 ceph-mon[76349]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:19 smithi184 ceph-mon[100622]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:19 smithi184 ceph-mon[102844]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:19 smithi196 ceph-mon[76349]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:20.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:20.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:20.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:21 smithi184 ceph-mon[100622]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:21 smithi184 ceph-mon[102844]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:21 smithi196 ceph-mon[76349]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:40:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:23 smithi184 ceph-mon[100622]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:23 smithi184 ceph-mon[102844]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:23 smithi196 ceph-mon[76349]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:23.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:23.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:23.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[100622]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[102844]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:24.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:24 smithi196 ceph-mon[76349]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:27 smithi196 ceph-mon[76349]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:27 smithi184 ceph-mon[100622]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:27 smithi184 ceph-mon[102844]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:28 smithi196 ceph-mon[76349]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:28 smithi184 ceph-mon[100622]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:28 smithi184 ceph-mon[102844]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:40:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:29.935 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:40:30.467 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 117m - - 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 116m - - 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (111m) 6s ago 112m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (112m) 5s ago 112m 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (105m) 6s ago 110m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (105m) 5s ago 116m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:40:30.468 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (106m) 6s ago 118m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (104m) 6s ago 118m 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (103m) 5s ago 116m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (104m) 6s ago 116m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (112m) 6s ago 112m 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (111m) 5s ago 111m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:40:30.469 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (102m) 6s ago 116m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (101m) 6s ago 115m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (101m) 6s ago 115m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (100m) 6s ago 114m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (99m) 5s ago 114m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (98m) 5s ago 114m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:40:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (97m) 5s ago 113m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (96m) 5s ago 113m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (105m) 5s ago 112m 73.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (96m) 6s ago 110m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (96m) 5s ago 111m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (96m) 6s ago 110m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:40:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (96m) 5s ago 110m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:40:30.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[100622]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:30.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[100622]: from='client.51345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[100622]: from='client.51351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[102844]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:30.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[102844]: from='client.51345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-mon[102844]: from='client.51351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.631 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:30.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:30.631 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:30.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:30.631 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:30.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:30 smithi196 ceph-mon[76349]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:30 smithi196 ceph-mon[76349]: from='client.51345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:30 smithi196 ceph-mon[76349]: from='client.51351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:40:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:40:30.782 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:40:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:40:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:40:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:40:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:40:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:40:31.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:40:31.364 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:31.364 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:31.364 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:31 smithi196 ceph-mon[76349]: from='client.61325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3524956356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:31 smithi196 ceph-mon[76349]: from='client.61337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[100622]: from='client.61325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3524956356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[100622]: from='client.61337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[102844]: from='client.61325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3524956356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:31.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:31 smithi184 ceph-mon[102844]: from='client.61337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:40:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2293836929' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:32 smithi196 ceph-mon[76349]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:32.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2293836929' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:32.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:32 smithi184 ceph-mon[100622]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:32.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2293836929' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:32.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:32 smithi184 ceph-mon[102844]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:33.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:33.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:33.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:34 smithi196 ceph-mon[76349]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:34 smithi184 ceph-mon[100622]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:34 smithi184 ceph-mon[102844]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:36 smithi196 ceph-mon[76349]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:36 smithi184 ceph-mon[100622]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:36 smithi184 ceph-mon[102844]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:38 smithi196 ceph-mon[76349]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:38 smithi184 ceph-mon[100622]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:38 smithi184 ceph-mon[102844]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:40 smithi196 ceph-mon[76349]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:40 smithi184 ceph-mon[100622]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:40 smithi184 ceph-mon[102844]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:40.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:40.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:40.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:40:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:42 smithi196 ceph-mon[76349]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:42.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:42 smithi184 ceph-mon[100622]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:42.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:42 smithi184 ceph-mon[102844]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:43.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:43.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:43.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[100622]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[102844]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:44 smithi196 ceph-mon[76349]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:45.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[100622]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:46 smithi184 ceph-mon[102844]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:40:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:46 smithi196 ceph-mon[76349]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:48 smithi184 ceph-mon[100622]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:48 smithi184 ceph-mon[102844]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:48 smithi196 ceph-mon[76349]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:50 smithi196 ceph-mon[76349]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:50 smithi184 ceph-mon[100622]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:50 smithi184 ceph-mon[102844]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:50.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:50.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:50.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:51.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:40:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:40:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:52 smithi196 ceph-mon[76349]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:52 smithi184 ceph-mon[100622]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:52 smithi184 ceph-mon[102844]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:53.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:53.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:40:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:40:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:40:53.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:40:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:54 smithi196 ceph-mon[76349]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:54 smithi184 ceph-mon[100622]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:54 smithi184 ceph-mon[102844]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:56 smithi196 ceph-mon[76349]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:56 smithi184 ceph-mon[102844]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:56 smithi184 ceph-mon[100622]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:58 smithi196 ceph-mon[76349]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:58 smithi184 ceph-mon[100622]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:58 smithi184 ceph-mon[102844]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:00 smithi196 ceph-mon[76349]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:00 smithi184 ceph-mon[100622]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:00 smithi184 ceph-mon[102844]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:00.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:00.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:00.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:01.661 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:41:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 117m - - 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 117m - - 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (111m) 18s ago 113m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (112m) 16s ago 112m 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (105m) 18s ago 111m 62.7M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (105m) 16s ago 116m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:41:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (106m) 18s ago 118m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (105m) 18s ago 118m 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (104m) 16s ago 117m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (104m) 18s ago 117m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (112m) 18s ago 112m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (112m) 16s ago 112m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (103m) 18s ago 116m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (102m) 18s ago 116m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (101m) 18s ago 115m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (101m) 18s ago 115m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (99m) 16s ago 115m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (99m) 16s ago 114m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (97m) 16s ago 114m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (97m) 16s ago 113m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (106m) 16s ago 113m 73.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:41:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (96m) 18s ago 111m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:41:02.189 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (96m) 16s ago 111m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:41:02.189 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (96m) 18s ago 111m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:41:02.189 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (96m) 16s ago 111m 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:41:02.498 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:02.499 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:41:02.500 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:41:02.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:41:02.500 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:41:02.771 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:41:02.772 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:41:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:02 smithi196 ceph-mon[76349]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:02 smithi196 ceph-mon[76349]: from='client.61349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:02 smithi196 ceph-mon[76349]: from='client.61355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:02 smithi196 ceph-mon[76349]: from='client.51393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/509242034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[100622]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[100622]: from='client.61349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[100622]: from='client.61355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[100622]: from='client.51393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/509242034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[102844]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[102844]: from='client.61349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[102844]: from='client.61355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[102844]: from='client.51393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/509242034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:03.081 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:41:03.081 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:03.081 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:03.081 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:03.081 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:03.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-mon[100622]: from='client.51405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2477119510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:03.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-mon[102844]: from='client.51405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:03.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2477119510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:03 smithi196 ceph-mon[76349]: from='client.51405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2477119510' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:03.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:03.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:03.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[100622]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[102844]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:04 smithi196 ceph-mon[76349]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[100622]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[102844]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:06 smithi196 ceph-mon[76349]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:08.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:08 smithi184 ceph-mon[100622]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:08.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:08 smithi184 ceph-mon[102844]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:08 smithi196 ceph-mon[76349]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:10 smithi196 ceph-mon[76349]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:10 smithi184 ceph-mon[100622]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:10 smithi184 ceph-mon[102844]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:10.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:10.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:10.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:41:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:12 smithi196 ceph-mon[76349]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:12 smithi184 ceph-mon[100622]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:12 smithi184 ceph-mon[102844]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:13.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:13.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:13.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:14 smithi196 ceph-mon[76349]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:14 smithi184 ceph-mon[100622]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:14 smithi184 ceph-mon[102844]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:16 smithi196 ceph-mon[76349]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:16 smithi184 ceph-mon[100622]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:16 smithi184 ceph-mon[102844]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:18.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:18 smithi196 ceph-mon[76349]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:18 smithi184 ceph-mon[100622]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:18 smithi184 ceph-mon[102844]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:20 smithi196 ceph-mon[76349]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:20 smithi184 ceph-mon[100622]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:20 smithi184 ceph-mon[102844]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:20.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:20.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:20.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:41:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:22 smithi196 ceph-mon[76349]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:22 smithi184 ceph-mon[100622]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:22 smithi184 ceph-mon[102844]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:23.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:23.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:23.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[102844]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[100622]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:25 smithi196 ceph-mon[76349]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[100622]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[102844]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:26.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:26 smithi196 ceph-mon[76349]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:28.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:28 smithi184 ceph-mon[100622]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:28.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:28 smithi184 ceph-mon[102844]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:28 smithi196 ceph-mon[76349]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:30 smithi184 ceph-mon[100622]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:30 smithi184 ceph-mon[102844]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:30.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:30.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:30.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:30 smithi196 ceph-mon[76349]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:41:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:32 smithi184 ceph-mon[100622]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:32 smithi184 ceph-mon[102844]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:32 smithi196 ceph-mon[76349]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.376 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:41:33.912 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 118m - - 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 118m - - 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (112m) 9s ago 113m 30.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (113m) 8s ago 113m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (106m) 9s ago 111m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (106m) 8s ago 117m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (107m) 9s ago 119m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (105m) 9s ago 119m 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (104m) 8s ago 117m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (105m) 9s ago 117m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (113m) 9s ago 113m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:41:33.913 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (113m) 8s ago 113m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (103m) 9s ago 117m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (102m) 9s ago 116m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (102m) 9s ago 116m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (101m) 9s ago 115m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (100m) 8s ago 115m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (99m) 8s ago 115m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (98m) 8s ago 114m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (97m) 8s ago 114m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (106m) 8s ago 113m 73.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (97m) 9s ago 112m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (97m) 8s ago 112m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (97m) 9s ago 111m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:41:33.914 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (97m) 8s ago 111m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:41:34.223 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:41:34.224 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:41:34.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:33.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:34.225 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:33.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:34.225 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:33.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:41:34.493 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:41:34.494 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:41:34.494 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:41:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[100622]: from='client.61385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[100622]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[100622]: from='client.61391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[100622]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/977580707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[102844]: from='client.61385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[102844]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[102844]: from='client.61391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[102844]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/977580707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:34.810 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:41:34.810 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:34.810 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:34.811 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:34 smithi196 ceph-mon[76349]: from='client.61385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:34 smithi196 ceph-mon[76349]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:34 smithi196 ceph-mon[76349]: from='client.61391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:34 smithi196 ceph-mon[76349]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:35.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/977580707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:35 smithi184 ceph-mon[100622]: from='client.61409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3721982345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:35 smithi184 ceph-mon[102844]: from='client.61409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3721982345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:35 smithi196 ceph-mon[76349]: from='client.61409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3721982345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:36 smithi184 ceph-mon[100622]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:36 smithi184 ceph-mon[102844]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:37.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:36 smithi196 ceph-mon[76349]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:38 smithi184 ceph-mon[100622]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:38 smithi184 ceph-mon[102844]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:38 smithi196 ceph-mon[76349]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:40 smithi184 ceph-mon[100622]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:40 smithi184 ceph-mon[102844]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:40.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:40.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:40.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:40 smithi196 ceph-mon[76349]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T21:41:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:42 smithi184 ceph-mon[100622]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:42 smithi184 ceph-mon[102844]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:42 smithi196 ceph-mon[76349]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:43.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:43.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:43.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[100622]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[102844]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:44 smithi196 ceph-mon[76349]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:46 smithi196 ceph-mon[76349]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[100622]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[102844]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:41:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:48 smithi184 ceph-mon[100622]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:48 smithi184 ceph-mon[102844]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:48 smithi196 ceph-mon[76349]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:50.821 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:50.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:50.821 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:50.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:50.821 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:50.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:50 smithi196 ceph-mon[76349]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:50 smithi184 ceph-mon[100622]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:50 smithi184 ceph-mon[102844]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:41:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:52 smithi196 ceph-mon[76349]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:52 smithi184 ceph-mon[100622]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:52 smithi184 ceph-mon[102844]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:53.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:53.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:41:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:41:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:41:53.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:41:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:54 smithi196 ceph-mon[76349]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:54 smithi184 ceph-mon[100622]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:54 smithi184 ceph-mon[102844]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:56 smithi196 ceph-mon[76349]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:56 smithi184 ceph-mon[100622]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:56 smithi184 ceph-mon[102844]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:58 smithi196 ceph-mon[76349]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:58.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:41:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:58 smithi184 ceph-mon[100622]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:41:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:58 smithi184 ceph-mon[102844]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:41:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:00.846 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:00.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:00.846 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:00.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:00.846 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:00.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:00 smithi196 ceph-mon[76349]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:00 smithi184 ceph-mon[100622]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:00 smithi184 ceph-mon[102844]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:42:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:02 smithi196 ceph-mon[76349]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:02 smithi184 ceph-mon[100622]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:02 smithi184 ceph-mon[102844]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:03.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:03.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:03.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:05.097 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:42:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[100622]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[102844]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 118m - - 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 118m - - 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (113m) 1s ago 114m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (113m) 19s ago 113m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (107m) 1s ago 112m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (106m) 19s ago 117m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:42:05.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (107m) 1s ago 119m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (106m) 1s ago 119m 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (105m) 19s ago 118m 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (105m) 1s ago 118m 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (113m) 1s ago 113m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (113m) 19s ago 113m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (104m) 1s ago 117m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (103m) 1s ago 117m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (102m) 1s ago 116m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (102m) 1s ago 116m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (101m) 19s ago 116m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (100m) 19s ago 115m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (99m) 19s ago 115m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (98m) 19s ago 114m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:42:05.631 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (107m) 19s ago 114m 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:42:05.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (98m) 1s ago 112m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:42:05.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (97m) 19s ago 112m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:42:05.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (97m) 1s ago 112m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:42:05.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (97m) 19s ago 112m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:42:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:05 smithi196 ceph-mon[76349]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:42:05.949 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:42:06.216 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:42:06.217 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:42:06.528 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:42:06.529 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:06.529 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:06.529 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:06.529 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:06.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[100622]: from='client.61421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:06.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1925068635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:06.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[102844]: from='client.61421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:06.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1925068635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:06.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:06 smithi196 ceph-mon[76349]: from='client.61421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1925068635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[100622]: from='client.61427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[100622]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[100622]: from='client.61433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[100622]: from='client.61445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1737968907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[102844]: from='client.61427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[102844]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[102844]: from='client.61433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[102844]: from='client.61445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1737968907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:07 smithi196 ceph-mon[76349]: from='client.61427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:07 smithi196 ceph-mon[76349]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:07 smithi196 ceph-mon[76349]: from='client.61433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:07 smithi196 ceph-mon[76349]: from='client.61445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1737968907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:09.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:09 smithi184 ceph-mon[100622]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:09.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:09 smithi184 ceph-mon[102844]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:09 smithi196 ceph-mon[76349]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:10.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:10.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:10.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:10.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:10.712 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:10.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:11 smithi196 ceph-mon[76349]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:11 smithi184 ceph-mon[100622]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:11 smithi184 ceph-mon[102844]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:42:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:13 smithi196 ceph-mon[76349]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:13 smithi184 ceph-mon[100622]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:13 smithi184 ceph-mon[102844]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:13.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:13.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:13.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:15 smithi196 ceph-mon[76349]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:15 smithi184 ceph-mon[100622]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:15 smithi184 ceph-mon[102844]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:17 smithi196 ceph-mon[76349]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:17 smithi184 ceph-mon[100622]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:17 smithi184 ceph-mon[102844]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:18.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:18 smithi196 ceph-mon[76349]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:18 smithi184 ceph-mon[100622]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:18 smithi184 ceph-mon[102844]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:20 smithi196 ceph-mon[76349]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:20 smithi184 ceph-mon[100622]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:20 smithi184 ceph-mon[102844]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:20.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:20.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:20.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:42:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:22 smithi196 ceph-mon[76349]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:22 smithi184 ceph-mon[100622]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:22 smithi184 ceph-mon[102844]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:23.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:23.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:23.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[102844]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[100622]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:24 smithi196 ceph-mon[76349]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:27 smithi196 ceph-mon[76349]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[100622]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[102844]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:29 smithi184 ceph-mon[102844]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:29 smithi184 ceph-mon[100622]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:29 smithi196 ceph-mon[76349]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:30.734 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:30.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:30.734 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:30.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:30.734 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:30.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:31 smithi184 ceph-mon[100622]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:31 smithi184 ceph-mon[102844]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:31 smithi196 ceph-mon[76349]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:42:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:33.384 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:33 smithi184 ceph-mon[100622]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:33.384 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:33 smithi184 ceph-mon[102844]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:33 smithi196 ceph-mon[76349]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:33.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:33.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:33.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:35 smithi196 ceph-mon[76349]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:35 smithi184 ceph-mon[100622]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:35 smithi184 ceph-mon[102844]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:36.825 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 119m - - 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 119m - - 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (113m) 12s ago 114m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (114m) 11s ago 114m 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (107m) 12s ago 112m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (107m) 11s ago 118m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (108m) 12s ago 2h 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:42:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (106m) 12s ago 2h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (106m) 11s ago 118m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (106m) 12s ago 118m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (114m) 12s ago 114m 16.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (114m) 11s ago 114m 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (104m) 12s ago 118m 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (104m) 12s ago 117m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (103m) 12s ago 117m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (102m) 12s ago 116m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (101m) 11s ago 116m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (100m) 11s ago 116m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (99m) 11s ago 115m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (98m) 11s ago 115m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (107m) 11s ago 114m 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (98m) 12s ago 113m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:42:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (98m) 11s ago 113m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:42:37.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (98m) 12s ago 112m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:42:37.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (98m) 11s ago 112m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:42:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:37 smithi196 ceph-mon[76349]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:37 smithi184 ceph-mon[102844]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:37 smithi184 ceph-mon[100622]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:42:37.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:42:37.666 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:42:37.935 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:42:37.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:42:38.249 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:42:38.249 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:38.249 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:38.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[100622]: from='client.61457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[100622]: from='client.51495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2457070637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:38.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[102844]: from='client.61457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[102844]: from='client.51495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2457070637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:38 smithi196 ceph-mon[76349]: from='client.61457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:38 smithi196 ceph-mon[76349]: from='client.51495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2457070637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:39 smithi196 ceph-mon[76349]: from='client.61469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:39 smithi196 ceph-mon[76349]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:39 smithi196 ceph-mon[76349]: from='client.61481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1759291131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[100622]: from='client.61469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[100622]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[100622]: from='client.61481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1759291131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[102844]: from='client.61469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[102844]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[102844]: from='client.61481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1759291131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:40.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:40.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:40.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:41.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:41 smithi196 ceph-mon[76349]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:41 smithi184 ceph-mon[100622]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:41 smithi184 ceph-mon[102844]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T21:42:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:43 smithi184 ceph-mon[100622]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:43 smithi184 ceph-mon[102844]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:43 smithi196 ceph-mon[76349]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:43.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:43.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:43.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:45 smithi196 ceph-mon[76349]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[100622]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[102844]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[100622]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[102844]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:47 smithi196 ceph-mon[76349]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:42:49.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:49 smithi184 ceph-mon[100622]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:49.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:49 smithi184 ceph-mon[102844]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:49 smithi196 ceph-mon[76349]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:50.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:50.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:50.755 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:50.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:50.755 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:50.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:51 smithi184 ceph-mon[100622]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:51 smithi184 ceph-mon[102844]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:51 smithi196 ceph-mon[76349]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:52.221 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:42:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:42:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:53 smithi184 ceph-mon[102844]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:53 smithi184 ceph-mon[100622]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:53 smithi196 ceph-mon[76349]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:53.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:53.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:42:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:42:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:42:53.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:42:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:55 smithi184 ceph-mon[100622]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:55 smithi184 ceph-mon[102844]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:55 smithi196 ceph-mon[76349]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:57 smithi184 ceph-mon[100622]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:57 smithi184 ceph-mon[102844]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:57 smithi196 ceph-mon[76349]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:59 smithi184 ceph-mon[100622]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:59 smithi184 ceph-mon[102844]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:59 smithi196 ceph-mon[76349]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:00.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:00.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:00.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:01 smithi196 ceph-mon[76349]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:01 smithi184 ceph-mon[100622]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:01 smithi184 ceph-mon[102844]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:43:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:03 smithi184 ceph-mon[100622]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:03 smithi184 ceph-mon[102844]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:03 smithi196 ceph-mon[76349]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:03.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:03.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:03.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[100622]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[102844]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:05 smithi196 ceph-mon[76349]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[100622]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[102844]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:07 smithi196 ceph-mon[76349]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:08.546 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 119m - - 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 119m - - 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (114m) 4s ago 115m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (114m) 2s ago 114m 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (108m) 4s ago 113m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:43:09.072 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (107m) 2s ago 118m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (108m) 4s ago 2h 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (107m) 4s ago 2h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (106m) 2s ago 119m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (106m) 4s ago 119m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (114m) 4s ago 114m 15.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (114m) 2s ago 114m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (105m) 4s ago 118m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (104m) 4s ago 118m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (103m) 4s ago 117m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (103m) 4s ago 117m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (102m) 2s ago 117m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (101m) 2s ago 116m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (100m) 2s ago 116m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (99m) 2s ago 116m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:43:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (108m) 2s ago 115m 75.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:43:09.074 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (99m) 4s ago 113m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:43:09.074 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (98m) 2s ago 113m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:43:09.074 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (98m) 4s ago 113m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:43:09.074 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (98m) 2s ago 113m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:43:09.383 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:43:09.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:09.385 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:43:09.385 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:43:09.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:43:09.385 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:43:09.385 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:09 smithi184 ceph-mon[102844]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:09.385 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:09 smithi184 ceph-mon[100622]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:09 smithi196 ceph-mon[76349]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:09.657 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:43:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:43:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:43:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:43:09.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:43:09.978 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:43:09.978 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:09.978 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[100622]: from='client.51525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[100622]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[100622]: from='client.61505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3724755003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/784898748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:10.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[102844]: from='client.51525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[102844]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[102844]: from='client.61505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3724755003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:10.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/784898748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:10 smithi196 ceph-mon[76349]: from='client.51525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:10 smithi196 ceph-mon[76349]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:10 smithi196 ceph-mon[76349]: from='client.61505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3724755003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/784898748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:10.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:10.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:10.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:10.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:10.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:10.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:11.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:11 smithi196 ceph-mon[76349]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:11.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:11 smithi196 ceph-mon[76349]: from='client.61517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:11 smithi184 ceph-mon[102844]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:11 smithi184 ceph-mon[102844]: from='client.61517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:11 smithi184 ceph-mon[100622]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:11 smithi184 ceph-mon[100622]: from='client.61517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:12.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:43:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:13 smithi184 ceph-mon[100622]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:13 smithi184 ceph-mon[102844]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:13 smithi196 ceph-mon[76349]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:14.270 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:13.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:14.271 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:13.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:14.271 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:13.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:14.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:15 smithi184 ceph-mon[100622]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:15 smithi184 ceph-mon[102844]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:15 smithi196 ceph-mon[76349]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:17 smithi184 ceph-mon[100622]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:17 smithi184 ceph-mon[102844]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:17 smithi196 ceph-mon[76349]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:19 smithi184 ceph-mon[100622]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:19 smithi184 ceph-mon[102844]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:19 smithi196 ceph-mon[76349]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:20.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:20.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:20.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:21 smithi184 ceph-mon[100622]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:21 smithi184 ceph-mon[102844]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:21 smithi196 ceph-mon[76349]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:22.288 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:43:22.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:23 smithi184 ceph-mon[100622]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:23 smithi184 ceph-mon[102844]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:23 smithi196 ceph-mon[76349]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:23.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:23.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:23.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[102844]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[100622]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:25 smithi196 ceph-mon[76349]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[102844]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[100622]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:27 smithi196 ceph-mon[76349]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:29 smithi196 ceph-mon[76349]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:29 smithi184 ceph-mon[102844]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:29 smithi184 ceph-mon[100622]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:30.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:30.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:30.795 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:30.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:31.565 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:31 smithi184 ceph-mon[100622]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:31.565 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:31 smithi184 ceph-mon[102844]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:31 smithi196 ceph-mon[76349]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:32.089 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:43:32.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:32.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:33 smithi184 ceph-mon[102844]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:33 smithi184 ceph-mon[100622]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:33 smithi196 ceph-mon[76349]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:33.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:33.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:33.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:35 smithi196 ceph-mon[76349]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:35 smithi184 ceph-mon[100622]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:35 smithi184 ceph-mon[102844]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:37 smithi196 ceph-mon[76349]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:37 smithi184 ceph-mon[100622]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:37 smithi184 ceph-mon[102844]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:39 smithi196 ceph-mon[76349]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:39 smithi184 ceph-mon[100622]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:39 smithi184 ceph-mon[102844]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:40.275 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (114m) 16s ago 115m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (115m) 14s ago 115m 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (108m) 16s ago 113m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (108m) 14s ago 119m 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (109m) 16s ago 2h 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (107m) 16s ago 2h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:43:40.804 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (107m) 14s ago 119m 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (107m) 16s ago 119m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (115m) 16s ago 115m 15.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (115m) 14s ago 115m 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (105m) 16s ago 119m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (105m) 16s ago 118m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (104m) 16s ago 118m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (103m) 16s ago 118m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (102m) 14s ago 117m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (101m) 14s ago 117m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (100m) 14s ago 116m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (99m) 14s ago 116m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (108m) 14s ago 116m 75.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (99m) 16s ago 114m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (99m) 14s ago 114m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:43:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (99m) 16s ago 113m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:43:40.806 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (99m) 14s ago 113m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:43:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:40.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:40.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:40.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:43:41.115 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:43:41.116 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:43:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:41 smithi196 ceph-mon[76349]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:41 smithi196 ceph-mon[76349]: from='client.51561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:41.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2333900337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:41.387 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:43:41.387 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:43:41.387 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:43:41.388 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[102844]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[102844]: from='client.51561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2333900337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[100622]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[100622]: from='client.51561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:41.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2333900337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:41.697 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:43:41.698 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:41.698 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:41.698 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:43:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:42 smithi196 ceph-mon[76349]: from='client.51567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:42 smithi196 ceph-mon[76349]: from='client.61541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:42.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2466840372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[100622]: from='client.51567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[100622]: from='client.61541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2466840372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[102844]: from='client.51567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[102844]: from='client.61541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2466840372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:43.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:43 smithi196 ceph-mon[76349]: from='client.61553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:43.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:43 smithi196 ceph-mon[76349]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-mon[100622]: from='client.61553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-mon[100622]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-mon[102844]: from='client.61553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-mon[102844]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:43.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:43.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:43.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:45 smithi196 ceph-mon[76349]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[100622]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[102844]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:46.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:46 smithi196 ceph-mon[76349]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:46 smithi184 ceph-mon[100622]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:46 smithi184 ceph-mon[102844]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:43:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:48 smithi196 ceph-mon[76349]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:48 smithi184 ceph-mon[100622]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:48 smithi184 ceph-mon[102844]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:50 smithi196 ceph-mon[76349]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:50 smithi184 ceph-mon[100622]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:50 smithi184 ceph-mon[102844]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:50.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:50.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:50.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:52.007 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:52.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:43:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:43:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:52 smithi196 ceph-mon[76349]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:52 smithi184 ceph-mon[100622]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:52 smithi184 ceph-mon[102844]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:53.993Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:53.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:43:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:43:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:43:53.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:43:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:54 smithi196 ceph-mon[76349]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:54 smithi184 ceph-mon[100622]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:54 smithi184 ceph-mon[102844]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:56 smithi196 ceph-mon[76349]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:56 smithi184 ceph-mon[100622]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:56 smithi184 ceph-mon[102844]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:58 smithi184 ceph-mon[100622]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:58 smithi184 ceph-mon[102844]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:58 smithi196 ceph-mon[76349]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:00.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:00 smithi184 ceph-mon[100622]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:00.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:00 smithi184 ceph-mon[102844]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:00.817 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:00.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:00.817 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:00.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:00.817 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:00.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:00 smithi196 ceph-mon[76349]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:01.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:44:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:02 smithi184 ceph-mon[100622]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:02 smithi184 ceph-mon[102844]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:02 smithi196 ceph-mon[76349]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:03.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:03.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:03 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:03.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:04 smithi196 ceph-mon[76349]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:04 smithi184 ceph-mon[100622]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:04 smithi184 ceph-mon[102844]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:07 smithi196 ceph-mon[76349]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:44:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[100622]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:44:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[102844]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:09 smithi196 ceph-mon[76349]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:09 smithi184 ceph-mon[100622]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:09 smithi184 ceph-mon[102844]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:10.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:10.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:10.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:11 smithi196 ceph-mon[76349]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:44:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:11 smithi184 ceph-mon[102844]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:44:11.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:11 smithi184 ceph-mon[100622]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:44:11.996 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:44:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:44:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:12.540 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (115m) 7s ago 116m 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (116m) 5s ago 116m 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (109m) 7s ago 114m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (108m) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (109m) 7s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (108m) 7s ago 2h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (107m) 5s ago 2h 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (107m) 7s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (115m) 7s ago 115m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (115m) 5s ago 115m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:44:12.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (106m) 7s ago 119m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (105m) 7s ago 119m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (104m) 7s ago 118m 64.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (104m) 7s ago 118m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (103m) 5s ago 118m 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (102m) 5s ago 117m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (101m) 5s ago 117m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (100m) 5s ago 117m 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (109m) 5s ago 116m 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (100m) 7s ago 114m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (99m) 5s ago 114m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (100m) 7s ago 114m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:44:12.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (99m) 5s ago 114m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:44:12.865 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:44:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:44:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:44:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:44:12.866 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:44:13.145 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:44:13.145 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:44:13.145 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:44:13.145 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:44:13.146 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:44:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:13 smithi196 ceph-mon[76349]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:44:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:13 smithi196 ceph-mon[76349]: from='client.61565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:13 smithi196 ceph-mon[76349]: from='client.51603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/291263626' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:13.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[100622]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[100622]: from='client.61565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[100622]: from='client.51603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/291263626' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[102844]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[102844]: from='client.61565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[102844]: from='client.51603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:13.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/291263626' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:13.478 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:44:13.478 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:13.478 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[102844]: from='client.51609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[102844]: from='client.61589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1753960015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[100622]: from='client.51609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[100622]: from='client.61589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1753960015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:13.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:13.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:13 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:13.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:14 smithi196 ceph-mon[76349]: from='client.51609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:14 smithi196 ceph-mon[76349]: from='client.61589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1753960015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:15 smithi184 ceph-mon[100622]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:15 smithi184 ceph-mon[102844]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:15 smithi196 ceph-mon[76349]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:17 smithi196 ceph-mon[76349]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-20T21:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:17 smithi184 ceph-mon[102844]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-20T21:44:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:17 smithi184 ceph-mon[100622]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-20T21:44:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:19 smithi196 ceph-mon[76349]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:44:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:19 smithi184 ceph-mon[102844]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:44:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:19 smithi184 ceph-mon[100622]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-20T21:44:20.828 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:20.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:20.829 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:20.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:20.829 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:20.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:21 smithi196 ceph-mon[76349]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T21:44:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:21 smithi184 ceph-mon[102844]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T21:44:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:21 smithi184 ceph-mon[100622]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T21:44:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:44:22.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:23 smithi196 ceph-mon[76349]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:44:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:23 smithi184 ceph-mon[102844]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:44:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:23 smithi184 ceph-mon[100622]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:44:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:23.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:23.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:23 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:23.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:25 smithi196 ceph-mon[76349]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:44:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:25 smithi184 ceph-mon[100622]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:44:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:25 smithi184 ceph-mon[102844]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:44:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[102844]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[100622]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:27 smithi196 ceph-mon[76349]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T21:44:27.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:27.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:29 smithi184 ceph-mon[102844]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T21:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:29 smithi184 ceph-mon[100622]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T21:44:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:29 smithi196 ceph-mon[76349]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T21:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:30.838 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:30.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:30.839 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:30.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:30.839 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:30.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:31 smithi184 ceph-mon[100622]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-20T21:44:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:31 smithi184 ceph-mon[102844]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-20T21:44:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:31 smithi196 ceph-mon[76349]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-20T21:44:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:32] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:44:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:33 smithi184 ceph-mon[100622]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:44:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:33 smithi184 ceph-mon[102844]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:44:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:33 smithi196 ceph-mon[76349]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:44:34.204 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:33.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:34.204 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:33.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:34.204 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:33 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:33.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:35 smithi184 ceph-mon[100622]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:44:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:35 smithi184 ceph-mon[102844]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:44:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:35 smithi196 ceph-mon[76349]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:44:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:37 smithi184 ceph-mon[100622]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:37 smithi184 ceph-mon[102844]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:37 smithi196 ceph-mon[76349]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:44:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:39 smithi184 ceph-mon[100622]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:44:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:39 smithi184 ceph-mon[102844]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:44:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:39 smithi196 ceph-mon[76349]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:44:40.842 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:40.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:40.842 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:40.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:40.842 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:40.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:41 smithi184 ceph-mon[100622]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:44:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:41 smithi184 ceph-mon[102844]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:44:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:41 smithi196 ceph-mon[76349]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T21:44:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:42] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:44:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:43 smithi184 ceph-mon[100622]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:43 smithi184 ceph-mon[102844]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:43.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:43 smithi196 ceph-mon[76349]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T21:44:43.785 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:44:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:43.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:43.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:43 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:43.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:44.332 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:44:44.332 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T21:44:44.332 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (115m) 19s ago 116m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (116m) 17s ago 116m 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (109m) 19s ago 114m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (109m) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (110m) 19s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (108m) 19s ago 2h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (108m) 17s ago 2h 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (108m) 19s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (116m) 19s ago 116m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (116m) 17s ago 116m 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (107m) 19s ago 2h 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:44:44.333 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (106m) 19s ago 119m 73.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (105m) 19s ago 119m 65.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (104m) 19s ago 119m 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (103m) 17s ago 118m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (102m) 17s ago 118m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (101m) 17s ago 117m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (101m) 17s ago 117m 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (109m) 17s ago 117m 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (100m) 19s ago 115m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (100m) 17s ago 115m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (100m) 19s ago 114m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:44:44.334 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (100m) 17s ago 114m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:44:44.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:44.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:44:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:44:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:44:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:44:44.641 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:44:44.920 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:44:44.920 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:44:44.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:44:45.234 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:44:45.235 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:45.235 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:45.235 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:45.235 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:45.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:45.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: from='client.51633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: from='client.51645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4253391181' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:45.430 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4257348248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: from='client.51633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: from='client.51645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4253391181' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4257348248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: from='client.51633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: from='client.51645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4253391181' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4257348248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:46 smithi196 ceph-mon[76349]: from='client.51657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[100622]: from='client.51657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[102844]: from='client.51657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:46.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:47 smithi196 ceph-mon[76349]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[100622]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[102844]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:49 smithi196 ceph-mon[76349]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:49 smithi184 ceph-mon[100622]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:49 smithi184 ceph-mon[102844]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:50.853 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:50.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:50.853 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:50.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:50.853 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:50.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:51 smithi196 ceph-mon[76349]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:51 smithi184 ceph-mon[100622]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:51 smithi184 ceph-mon[102844]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:44:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:44:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:53.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:53 smithi184 ceph-mon[100622]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:53.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:53 smithi184 ceph-mon[102844]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:53 smithi196 ceph-mon[76349]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:53.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:53 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:53.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:44:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:44:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:44:53.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:44:55.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:55 smithi184 ceph-mon[100622]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:55.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:55 smithi184 ceph-mon[102844]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:55 smithi196 ceph-mon[76349]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:57 smithi196 ceph-mon[76349]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:57 smithi184 ceph-mon[102844]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:57 smithi184 ceph-mon[100622]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:58 smithi196 ceph-mon[76349]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:58 smithi184 ceph-mon[100622]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:58 smithi184 ceph-mon[102844]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:44:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:44:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:44:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:00 smithi196 ceph-mon[76349]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:00 smithi184 ceph-mon[100622]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:00 smithi184 ceph-mon[102844]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:00.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:00.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:00.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:45:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:02 smithi184 ceph-mon[100622]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:02 smithi184 ceph-mon[102844]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:02 smithi196 ceph-mon[76349]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:03.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:04.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:04.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:05.031 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:04 smithi196 ceph-mon[76349]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:04 smithi184 ceph-mon[100622]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:04 smithi184 ceph-mon[102844]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[100622]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:06 smithi184 ceph-mon[102844]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:06.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:06 smithi196 ceph-mon[76349]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:09 smithi196 ceph-mon[76349]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:09 smithi184 ceph-mon[100622]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:09 smithi184 ceph-mon[102844]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:10.869 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:10.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:10.869 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:10.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:10.869 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:10.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:11 smithi184 ceph-mon[100622]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:11 smithi184 ceph-mon[102844]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:11 smithi196 ceph-mon[76349]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:45:12.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:13 smithi196 ceph-mon[76349]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:13 smithi184 ceph-mon[100622]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:13 smithi184 ceph-mon[102844]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:14.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:14.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:14.269 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:14.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:14.269 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:14.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:14.269 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:14.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:15 smithi196 ceph-mon[76349]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:15.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:15 smithi184 ceph-mon[100622]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:15.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:15 smithi184 ceph-mon[102844]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:15.547 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 2h - - 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 2h - - 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (116m) 10s ago 117m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (117m) 8s ago 117m 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (110m) 10s ago 115m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (110m) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (110m) 10s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:45:16.096 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (109m) 10s ago 2h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (108m) 8s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (108m) 10s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (116m) 10s ago 116m 15.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (116m) 8s ago 116m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (107m) 10s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (106m) 10s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (105m) 10s ago 2h 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (105m) 10s ago 119m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (104m) 8s ago 119m 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (103m) 8s ago 118m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (102m) 8s ago 118m 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (101m) 8s ago 118m 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (110m) 8s ago 117m 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (101m) 10s ago 115m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (100m) 8s ago 115m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:45:16.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (101m) 10s ago 115m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:45:16.098 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (100m) 8s ago 115m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:45:16.417 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:45:16.417 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:45:16.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:45:16.419 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:45:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:45:16.690 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:45:16.996 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:45:16.996 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:16.996 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:16.996 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:16.996 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: from='client.51669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: from='client.51675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: from='client.61649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4264718025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3245130955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: from='client.51669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: from='client.51675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: from='client.61649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4264718025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3245130955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: from='client.51669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: from='client.51675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: from='client.61649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4264718025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:17.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3245130955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:18 smithi184 ceph-mon[100622]: from='client.61661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:18 smithi184 ceph-mon[102844]: from='client.61661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:18 smithi196 ceph-mon[76349]: from='client.61661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:19 smithi184 ceph-mon[100622]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:19 smithi184 ceph-mon[102844]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:19 smithi196 ceph-mon[76349]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:20.880 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:20.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:20.880 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:20.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:20.880 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:20.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:21 smithi184 ceph-mon[100622]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:21 smithi184 ceph-mon[102844]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:21 smithi196 ceph-mon[76349]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:45:22.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:23 smithi184 ceph-mon[100622]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:23 smithi184 ceph-mon[102844]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:23 smithi196 ceph-mon[76349]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:24.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:24.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:24.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:25.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:25 smithi196 ceph-mon[76349]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:25 smithi184 ceph-mon[100622]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:25 smithi184 ceph-mon[102844]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:26.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:26.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:26.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:26.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:26 smithi196 ceph-mon[76349]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[102844]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:26 smithi184 ceph-mon[100622]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:29 smithi184 ceph-mon[100622]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:29 smithi184 ceph-mon[102844]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:29 smithi196 ceph-mon[76349]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:30.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:30.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:30.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:30.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:30.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:30.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:31 smithi184 ceph-mon[100622]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:31 smithi184 ceph-mon[102844]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:31 smithi196 ceph-mon[76349]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:45:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:33.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:33 smithi184 ceph-mon[100622]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:33.504 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:33 smithi184 ceph-mon[102844]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:33 smithi196 ceph-mon[76349]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:34.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:34.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:34.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:35.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:35 smithi184 ceph-mon[100622]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:35.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:35 smithi184 ceph-mon[102844]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:35 smithi196 ceph-mon[76349]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:37.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:37 smithi196 ceph-mon[76349]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:37 smithi184 ceph-mon[100622]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:37 smithi184 ceph-mon[102844]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:39 smithi184 ceph-mon[100622]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:39 smithi184 ceph-mon[102844]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:39 smithi196 ceph-mon[76349]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:40.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:40.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:40.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:40.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:40.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:40.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:41 smithi184 ceph-mon[100622]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:41 smithi184 ceph-mon[102844]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:41 smithi196 ceph-mon[76349]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:42.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:45:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:42.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:43 smithi184 ceph-mon[102844]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:43 smithi184 ceph-mon[100622]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:43 smithi196 ceph-mon[76349]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:44.269 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:44.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:44.270 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:44.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:44.270 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:44.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:44.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:45 smithi184 ceph-mon[100622]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:45 smithi184 ceph-mon[102844]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:45.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:45 smithi196 ceph-mon[76349]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[100622]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[102844]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:46 smithi196 ceph-mon[76349]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:47.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:47.293 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:45:47.827 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 2h - - 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (116m) 2s ago 117m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (117m) 0s ago 117m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (110m) 2s ago 115m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (110m) 0s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (111m) 2s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (109m) 2s ago 2h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (109m) 0s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (109m) 2s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:45:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (117m) 2s ago 117m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (117m) 0s ago 117m 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (108m) 2s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (107m) 2s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (106m) 2s ago 2h 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (106m) 2s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (104m) 0s ago 119m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (103m) 0s ago 119m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (102m) 0s ago 119m 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (102m) 0s ago 118m 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (110m) 0s ago 118m 75.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (101m) 2s ago 116m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (101m) 0s ago 116m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:45:47.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (101m) 2s ago 115m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:45:47.830 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (101m) 0s ago 115m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:45:48.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:45:48.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:45:48.421 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:45:48.421 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:45:48.421 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:45:48.421 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:45:48.421 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:45:48.422 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[100622]: from='client.61673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1428033292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[102844]: from='client.61673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1428033292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:48 smithi196 ceph-mon[76349]: from='client.61673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1428033292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:48.731 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:45:48.732 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:48.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:48.732 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:48.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:49 smithi196 ceph-mon[76349]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:49 smithi196 ceph-mon[76349]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:49 smithi196 ceph-mon[76349]: from='client.61685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/889498418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[100622]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[100622]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[100622]: from='client.61685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/889498418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[102844]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[102844]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:49.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[102844]: from='client.61685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:49.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/889498418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:50 smithi196 ceph-mon[76349]: from='client.51729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:50 smithi184 ceph-mon[100622]: from='client.51729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:50 smithi184 ceph-mon[102844]: from='client.51729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:50.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:50.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:50.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:51 smithi196 ceph-mon[76349]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:51 smithi184 ceph-mon[100622]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:51 smithi184 ceph-mon[102844]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:45:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:45:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:52 smithi196 ceph-mon[76349]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:52 smithi184 ceph-mon[100622]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:52 smithi184 ceph-mon[102844]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:54.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:54.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:45:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:45:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:45:54.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:45:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:54 smithi184 ceph-mon[100622]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:54 smithi184 ceph-mon[102844]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:54 smithi196 ceph-mon[76349]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:56.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:56 smithi184 ceph-mon[100622]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:56.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:56 smithi184 ceph-mon[102844]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:56 smithi196 ceph-mon[76349]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:58 smithi184 ceph-mon[100622]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:58 smithi184 ceph-mon[102844]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:58 smithi196 ceph-mon[76349]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:00.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:00 smithi184 ceph-mon[100622]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:00.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:00 smithi184 ceph-mon[102844]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:00.905 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:00.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:00.905 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:00.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:00.905 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:00.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:00 smithi196 ceph-mon[76349]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:02.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:46:03.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:02 smithi184 ceph-mon[100622]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:02 smithi184 ceph-mon[102844]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:02 smithi196 ceph-mon[76349]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:04.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:04.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:04.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:04 smithi184 ceph-mon[100622]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:04 smithi184 ceph-mon[102844]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:04 smithi196 ceph-mon[76349]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:06 smithi196 ceph-mon[76349]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[100622]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[102844]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:08 smithi196 ceph-mon[76349]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[100622]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:08 smithi184 ceph-mon[102844]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:10.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:10 smithi184 ceph-mon[100622]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:10.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:10 smithi184 ceph-mon[102844]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:10.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:10.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:10.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:10.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:10.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:10.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:10 smithi196 ceph-mon[76349]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:46:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:12 smithi184 ceph-mon[100622]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:12 smithi184 ceph-mon[102844]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:12 smithi196 ceph-mon[76349]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:14.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:14.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:14.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:14.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:14.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:14.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-mon[100622]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-mon[102844]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:14 smithi196 ceph-mon[76349]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:17.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:16 smithi184 ceph-mon[100622]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:17.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:16 smithi184 ceph-mon[102844]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:16 smithi196 ceph-mon[76349]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:18 smithi196 ceph-mon[76349]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:18.931 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:18 smithi184 ceph-mon[100622]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:18.932 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:18 smithi184 ceph-mon[102844]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:19.030 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:46:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:46:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 2h - - 2024-09-20T21:46:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T21:46:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (117m) 13s ago 118m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:46:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (118m) 12s ago 118m 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (111m) 13s ago 116m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (111m) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (111m) 13s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (110m) 13s ago 2h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (109m) 12s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (110m) 13s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (117m) 13s ago 117m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (117m) 12s ago 117m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (108m) 13s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (107m) 13s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (107m) 13s ago 2h 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (106m) 13s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (105m) 12s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:46:19.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (104m) 12s ago 119m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (103m) 12s ago 119m 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (102m) 12s ago 119m 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (111m) 12s ago 118m 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (102m) 13s ago 116m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (102m) 12s ago 116m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (102m) 13s ago 116m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:46:19.574 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (101m) 12s ago 116m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:19.885 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:46:19.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:46:19.887 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:46:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:19 smithi184 ceph-mon[102844]: from='client.61709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:19 smithi184 ceph-mon[102844]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:19 smithi184 ceph-mon[100622]: from='client.61709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:19 smithi184 ceph-mon[100622]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:19 smithi196 ceph-mon[76349]: from='client.61709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:19 smithi196 ceph-mon[76349]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:20.153 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:46:20.153 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:46:20.153 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:46:20.154 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:46:20.464 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:46:20.464 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:20.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:20.464 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:20.465 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:20.735 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:20.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:20.735 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:20.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:20.735 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:20.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[102844]: from='client.61721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[102844]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1734487232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[102844]: from='client.51765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3786270849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[100622]: from='client.61721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[100622]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1734487232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[100622]: from='client.51765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3786270849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:20 smithi196 ceph-mon[76349]: from='client.61721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:20 smithi196 ceph-mon[76349]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1734487232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:20 smithi196 ceph-mon[76349]: from='client.51765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3786270849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:46:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:22 smithi184 ceph-mon[100622]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:22 smithi184 ceph-mon[102844]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:22 smithi196 ceph-mon[76349]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:24.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:24.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:24.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:24 smithi184 ceph-mon[100622]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:24 smithi184 ceph-mon[102844]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:24 smithi196 ceph-mon[76349]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:27.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:26 smithi196 ceph-mon[76349]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:27.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[100622]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[102844]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:28 smithi196 ceph-mon[76349]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[100622]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:28.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:28 smithi184 ceph-mon[102844]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:30 smithi196 ceph-mon[76349]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:30.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:30 smithi184 ceph-mon[100622]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:30.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:30 smithi184 ceph-mon[102844]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:30.934 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:30.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:30.934 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:30.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:30.934 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:30.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:46:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:32 smithi184 ceph-mon[100622]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:32 smithi184 ceph-mon[102844]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:32 smithi196 ceph-mon[76349]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:34.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:34.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:34.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:34 smithi184 ceph-mon[100622]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:34 smithi184 ceph-mon[102844]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:34 smithi196 ceph-mon[76349]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:36 smithi184 ceph-mon[100622]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:36 smithi184 ceph-mon[102844]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:36 smithi196 ceph-mon[76349]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:38 smithi184 ceph-mon[102844]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:38 smithi184 ceph-mon[100622]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:38 smithi196 ceph-mon[76349]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:40.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:40 smithi184 ceph-mon[100622]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:40.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:40 smithi184 ceph-mon[102844]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:40.940 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:40.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:40.940 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:40.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:40.940 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:40.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:40 smithi196 ceph-mon[76349]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:42.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:46:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:42 smithi184 ceph-mon[100622]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:42 smithi184 ceph-mon[102844]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:43.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:42 smithi196 ceph-mon[76349]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:44.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:44.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:44.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-mon[100622]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-mon[102844]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:44 smithi196 ceph-mon[76349]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:47 smithi196 ceph-mon[76349]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[100622]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[102844]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:48 smithi196 ceph-mon[76349]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[100622]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[102844]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:46:50.762 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:46:50.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:50 smithi184 ceph-mon[100622]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:50.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:50 smithi184 ceph-mon[102844]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:50.951 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:50.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:50.951 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:50.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:50.951 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:50.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:50 smithi196 ceph-mon[76349]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 2h - - 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (117m) 5s ago 118m 29.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (118m) 3s ago 118m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (111m) 5s ago 116m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (111m) 3s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (112m) 5s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (110m) 5s ago 2h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:46:51.288 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (110m) 3s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (110m) 5s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (118m) 5s ago 118m 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (118m) 3s ago 118m 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (109m) 5s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (108m) 5s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (107m) 5s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (107m) 5s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (105m) 3s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (104m) 3s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (103m) 3s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (103m) 3s ago 119m 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (111m) 3s ago 119m 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (102m) 5s ago 117m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (102m) 3s ago 117m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:46:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (102m) 5s ago 116m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:46:51.290 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (102m) 3s ago 116m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:46:51.599 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:46:51.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:46:51.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:46:51.872 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:46:51.873 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:46:51.874 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[100622]: from='client.61745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[100622]: from='client.51783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[100622]: from='client.51789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/57511132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[102844]: from='client.61745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:51.874 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[102844]: from='client.51783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.875 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[102844]: from='client.51789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:51.875 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/57511132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:51 smithi196 ceph-mon[76349]: from='client.61745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:51 smithi196 ceph-mon[76349]: from='client.51783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:51 smithi196 ceph-mon[76349]: from='client.51789 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/57511132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:52.183 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:46:52.183 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:52.183 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:52.183 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:52.183 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:52.184 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:46:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:46:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[100622]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[100622]: from='client.51801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3583938452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[102844]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[102844]: from='client.51801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3583938452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:52 smithi196 ceph-mon[76349]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:52 smithi196 ceph-mon[76349]: from='client.51801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3583938452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:54.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:54.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:46:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:46:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:46:54.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:46:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:54 smithi184 ceph-mon[100622]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:54 smithi184 ceph-mon[102844]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:54 smithi196 ceph-mon[76349]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:56 smithi184 ceph-mon[100622]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:56 smithi184 ceph-mon[102844]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:56 smithi196 ceph-mon[76349]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:58 smithi184 ceph-mon[100622]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:58 smithi184 ceph-mon[102844]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:58 smithi196 ceph-mon[76349]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:00.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:00.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:00.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:00.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:00.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:00.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:00 smithi184 ceph-mon[100622]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:00 smithi184 ceph-mon[102844]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:00 smithi196 ceph-mon[76349]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:47:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:02 smithi184 ceph-mon[100622]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:02 smithi184 ceph-mon[102844]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:02 smithi196 ceph-mon[76349]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:04.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:04.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:04.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:04 smithi184 ceph-mon[100622]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:04 smithi184 ceph-mon[102844]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:04 smithi196 ceph-mon[76349]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:07 smithi196 ceph-mon[76349]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[100622]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[102844]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:08 smithi196 ceph-mon[76349]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:09.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[100622]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[102844]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:10.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:10.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:10.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:10.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:10.860 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:10.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:10 smithi196 ceph-mon[76349]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:10 smithi184 ceph-mon[100622]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:10 smithi184 ceph-mon[102844]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:47:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:12 smithi196 ceph-mon[76349]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:12 smithi184 ceph-mon[100622]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:12 smithi184 ceph-mon[102844]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:14.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:14.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:14.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:14 smithi196 ceph-mon[76349]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:14 smithi184 ceph-mon[100622]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:14 smithi184 ceph-mon[102844]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:16 smithi196 ceph-mon[76349]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:16 smithi184 ceph-mon[100622]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:16 smithi184 ceph-mon[102844]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:18 smithi196 ceph-mon[76349]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:19.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:18 smithi184 ceph-mon[100622]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:19.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:18 smithi184 ceph-mon[102844]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:20.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:20.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:20.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:20 smithi196 ceph-mon[76349]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:21.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:20 smithi184 ceph-mon[100622]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:21.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:20 smithi184 ceph-mon[102844]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:47:22.478 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:47:23.007 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (118m) 16s ago 119m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (119m) 15s ago 119m 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (112m) 16s ago 117m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (112m) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (113m) 16s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (111m) 16s ago 2h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (110m) 15s ago 2h 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (111m) 16s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (118m) 16s ago 118m 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (118m) 15s ago 118m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (109m) 16s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:47:23.008 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (108m) 16s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (108m) 16s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (107m) 16s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (106m) 15s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (105m) 15s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (104m) 15s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (103m) 15s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (112m) 15s ago 119m 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (103m) 16s ago 117m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (103m) 15s ago 117m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (103m) 16s ago 117m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:47:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (103m) 15s ago 117m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:47:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:22 smithi184 ceph-mon[100622]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:22 smithi184 ceph-mon[102844]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:47:23.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:47:23.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:47:23.315 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:47:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:22 smithi196 ceph-mon[76349]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:47:23.587 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:47:23.588 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:47:23.898 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:47:23.898 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:23.898 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:23.898 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:23.898 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:23 smithi196 ceph-mon[76349]: from='client.51813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:23 smithi196 ceph-mon[76349]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:23 smithi196 ceph-mon[76349]: from='client.61793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1434228135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[102844]: from='client.51813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[102844]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[102844]: from='client.61793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1434228135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[100622]: from='client.51813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[100622]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[100622]: from='client.61793 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1434228135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:24.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:24.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:24.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:24.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[102844]: from='client.51837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[102844]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/282542780' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[100622]: from='client.51837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[100622]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/282542780' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:24 smithi196 ceph-mon[76349]: from='client.51837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:24 smithi196 ceph-mon[76349]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/282542780' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:27.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:27 smithi196 ceph-mon[76349]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:27.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[102844]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[100622]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:28 smithi196 ceph-mon[76349]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[100622]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[102844]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:30.979 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:30.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:30.979 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:30.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:30.980 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:30.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:30 smithi184 ceph-mon[100622]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:30 smithi184 ceph-mon[102844]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:30 smithi196 ceph-mon[76349]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:47:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:32 smithi184 ceph-mon[100622]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:32 smithi184 ceph-mon[102844]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:32 smithi196 ceph-mon[76349]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:34.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:34.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:34.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:34 smithi184 ceph-mon[100622]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:34 smithi184 ceph-mon[102844]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:34 smithi196 ceph-mon[76349]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:36 smithi184 ceph-mon[100622]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:36 smithi184 ceph-mon[102844]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:36 smithi196 ceph-mon[76349]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:39 smithi184 ceph-mon[100622]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:39 smithi184 ceph-mon[102844]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:39 smithi196 ceph-mon[76349]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:40.729 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:40.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:40.729 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:40.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:40.729 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:40.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:41.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:41 smithi184 ceph-mon[102844]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:41.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:41 smithi184 ceph-mon[100622]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:41 smithi196 ceph-mon[76349]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:42.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:47:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:43 smithi184 ceph-mon[100622]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:43 smithi184 ceph-mon[102844]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:43 smithi196 ceph-mon[76349]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:44.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:44.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:44.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:45 smithi184 ceph-mon[100622]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:45 smithi184 ceph-mon[102844]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:45 smithi196 ceph-mon[76349]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[100622]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[102844]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:47 smithi196 ceph-mon[76349]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:49 smithi196 ceph-mon[76349]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[100622]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[102844]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:47:50.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:50.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:50.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:50.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:50.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:50.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:51.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:51 smithi184 ceph-mon[100622]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:51.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:51 smithi184 ceph-mon[102844]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:51 smithi196 ceph-mon[76349]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:47:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:47:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:53 smithi196 ceph-mon[76349]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:53 smithi184 ceph-mon[100622]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:53 smithi184 ceph-mon[102844]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:54.199 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:47:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:54.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:54.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:47:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:47:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:47:54.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 2h - - 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (118m) 8s ago 119m 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (119m) 6s ago 119m 91.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (112m) 8s ago 117m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (112m) 6s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:47:54.728 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (113m) 8s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (111m) 8s ago 2h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (111m) 6s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (111m) 8s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (119m) 8s ago 119m 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (119m) 6s ago 119m 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (110m) 8s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (109m) 8s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (108m) 8s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (108m) 8s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (106m) 6s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (105m) 6s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (104m) 6s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (104m) 6s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:47:54.729 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (113m) 6s ago 2h 74.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:47:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (103m) 8s ago 118m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:47:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (103m) 6s ago 118m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:47:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (103m) 8s ago 118m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:47:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (103m) 6s ago 117m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:47:55.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:47:55.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:47:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[102844]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[102844]: from='client.51849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1771596171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[100622]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[100622]: from='client.51849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1771596171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:47:55.314 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:47:55.315 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:55 smithi196 ceph-mon[76349]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:55 smithi196 ceph-mon[76349]: from='client.51849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1771596171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:55.626 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:47:55.626 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:55.626 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:56 smithi196 ceph-mon[76349]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:56 smithi196 ceph-mon[76349]: from='client.61829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:56 smithi196 ceph-mon[76349]: from='client.61841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2052224796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[100622]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[100622]: from='client.61829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[100622]: from='client.61841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2052224796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[102844]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[102844]: from='client.61829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[102844]: from='client.61841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2052224796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:57 smithi196 ceph-mon[76349]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:57 smithi184 ceph-mon[100622]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:57 smithi184 ceph-mon[102844]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:59 smithi196 ceph-mon[76349]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:59 smithi184 ceph-mon[100622]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:59 smithi184 ceph-mon[102844]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:01.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:00.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:01.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:00.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:01.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:00.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:01 smithi184 ceph-mon[100622]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:01 smithi184 ceph-mon[102844]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:01 smithi196 ceph-mon[76349]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:48:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:03 smithi184 ceph-mon[100622]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:03 smithi184 ceph-mon[102844]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:03 smithi196 ceph-mon[76349]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:04.013Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:04.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:04.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:05 smithi184 ceph-mon[100622]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:05 smithi184 ceph-mon[102844]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:05 smithi196 ceph-mon[76349]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[100622]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[102844]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:07 smithi196 ceph-mon[76349]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.522 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:09 smithi196 ceph-mon[76349]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:09.522 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.522 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.522 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[100622]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[102844]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:11.012 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:10.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:11.013 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:10.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:11.013 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:10.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:11 smithi184 ceph-mon[100622]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:11 smithi184 ceph-mon[102844]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:11 smithi196 ceph-mon[76349]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:48:12.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:13 smithi184 ceph-mon[100622]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:13 smithi184 ceph-mon[102844]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:13 smithi196 ceph-mon[76349]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:48:13.813445085Z level=info msg="Completed cleanup jobs" duration=13.268132ms 2024-09-20T21:48:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:14.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:14.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:14.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:15 smithi184 ceph-mon[100622]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:15 smithi184 ceph-mon[102844]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:15 smithi196 ceph-mon[76349]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:17 smithi184 ceph-mon[100622]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:17 smithi184 ceph-mon[102844]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:17 smithi196 ceph-mon[76349]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:19 smithi184 ceph-mon[100622]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:19 smithi184 ceph-mon[102844]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:19 smithi196 ceph-mon[76349]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:20.904 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:20.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:20.904 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:20.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:20.904 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:20.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:21 smithi184 ceph-mon[100622]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:21 smithi184 ceph-mon[102844]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:21 smithi196 ceph-mon[76349]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:48:22.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:22.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:23 smithi184 ceph-mon[100622]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:23 smithi184 ceph-mon[102844]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:23 smithi196 ceph-mon[76349]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:24.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:24.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:24.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:25 smithi184 ceph-mon[100622]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:25 smithi184 ceph-mon[102844]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:25 smithi196 ceph-mon[76349]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:25.922 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:48:26.447 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (119m) 19s ago 2h 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (113m) 19s ago 118m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (113m) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (114m) 19s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (112m) 19s ago 2h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (111m) 18s ago 2h 160M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (112m) 19s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 119m 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (119m) 18s ago 119m 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:48:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (110m) 19s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (109m) 19s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (109m) 19s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (108m) 19s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (107m) 18s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (106m) 18s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (105m) 18s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (104m) 18s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (113m) 18s ago 2h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (104m) 19s ago 118m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (104m) 18s ago 119m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (104m) 19s ago 118m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:48:26.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (104m) 18s ago 118m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:48:26.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:48:26.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:48:27.036 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:48:27.036 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:48:27.036 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:48:27.037 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:48:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='client.51885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='client.51891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3344472450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='client.51885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='client.51891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3344472450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:27.347 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:48:27.347 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:27.347 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='client.51885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='client.51891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3344472450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[100622]: from='client.61865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[100622]: from='client.51909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3163684941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[102844]: from='client.61865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[102844]: from='client.51909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3163684941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:28 smithi196 ceph-mon[76349]: from='client.61865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:28 smithi196 ceph-mon[76349]: from='client.51909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3163684941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:29 smithi196 ceph-mon[76349]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[100622]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[102844]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:31.033 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:30.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:31.033 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:30.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:31.033 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:30.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:31.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:31 smithi196 ceph-mon[76349]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:31 smithi184 ceph-mon[100622]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:31 smithi184 ceph-mon[102844]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:48:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:33 smithi184 ceph-mon[100622]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:33 smithi184 ceph-mon[102844]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:33 smithi196 ceph-mon[76349]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:34.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:34.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:34.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:34 smithi184 ceph-mon[100622]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:34 smithi184 ceph-mon[102844]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:34 smithi196 ceph-mon[76349]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:36 smithi184 ceph-mon[100622]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:36 smithi184 ceph-mon[102844]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:36 smithi196 ceph-mon[76349]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:38 smithi184 ceph-mon[100622]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:38 smithi184 ceph-mon[102844]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:38 smithi196 ceph-mon[76349]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:41.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:40 smithi184 ceph-mon[100622]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:41.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:40 smithi184 ceph-mon[102844]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:41.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:40.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:41.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:40.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:41.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:40.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:40 smithi196 ceph-mon[76349]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:41.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:41.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:42.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:48:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:42 smithi184 ceph-mon[100622]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:42 smithi184 ceph-mon[102844]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:42 smithi196 ceph-mon[76349]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:44.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:44.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:44.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-mon[100622]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-mon[102844]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:44 smithi196 ceph-mon[76349]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:46 smithi184 ceph-mon[100622]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:46 smithi184 ceph-mon[102844]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:46 smithi196 ceph-mon[76349]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[102844]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[100622]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:48 smithi196 ceph-mon[76349]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:48:50.895 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:50.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:50.895 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:50.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:50.895 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:50.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:50.999 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:50 smithi196 ceph-mon[76349]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:50 smithi184 ceph-mon[100622]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:50 smithi184 ceph-mon[102844]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:48:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:48:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:52 smithi184 ceph-mon[100622]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:52 smithi184 ceph-mon[102844]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:52 smithi196 ceph-mon[76349]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:54.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:54.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:48:54.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:48:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:48:54.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:48:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:54 smithi184 ceph-mon[100622]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:54 smithi184 ceph-mon[102844]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:54 smithi196 ceph-mon[76349]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:56 smithi184 ceph-mon[100622]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:56 smithi184 ceph-mon[102844]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:56 smithi196 ceph-mon[76349]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:57.646 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:48:58.171 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 2h - - 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (119m) 11s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 93.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (113m) 11s ago 118m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (113m) 9s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (114m) 11s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (112m) 11s ago 2h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (112m) 9s ago 2h 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (112m) 11s ago 2h 168M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 16.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (111m) 11s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:48:58.172 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (110m) 11s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (109m) 11s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (109m) 11s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (107m) 9s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (107m) 9s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (105m) 9s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (105m) 9s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (114m) 9s ago 2h 72.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (104m) 11s ago 119m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (104m) 9s ago 119m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (104m) 11s ago 119m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:48:58.173 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (104m) 9s ago 119m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:48:58.480 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:48:58.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:48:58.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:48:58.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:48:58.481 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:48:58.749 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:48:58.750 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: from='client.51921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: from='client.61895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: from='client.61901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2254798558' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: from='client.51921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: from='client.61895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: from='client.61901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2254798558' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:59.065 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:48:59.066 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:59.066 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:59.066 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:59.066 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: from='client.51921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: from='client.61895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: from='client.61901 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2254798558' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:59 smithi184 ceph-mon[100622]: from='client.61913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:48:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3643625672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:59 smithi184 ceph-mon[102844]: from='client.61913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:48:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3643625672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:59 smithi196 ceph-mon[76349]: from='client.61913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:48:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3643625672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:00.918 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:00.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:00.918 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:00.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:00.918 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:00.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:00 smithi184 ceph-mon[100622]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:00 smithi184 ceph-mon[102844]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:00 smithi196 ceph-mon[76349]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:49:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:02 smithi184 ceph-mon[100622]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:02 smithi184 ceph-mon[102844]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:02 smithi196 ceph-mon[76349]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:04.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:04.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:04.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:04 smithi184 ceph-mon[100622]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:04 smithi184 ceph-mon[102844]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:04 smithi196 ceph-mon[76349]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:06 smithi184 ceph-mon[100622]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:06 smithi184 ceph-mon[102844]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:06 smithi196 ceph-mon[76349]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[100622]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[102844]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:09 smithi196 ceph-mon[76349]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:11.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:10.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:11.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:10.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:11.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:10.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:11 smithi184 ceph-mon[100622]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:11 smithi184 ceph-mon[102844]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:11 smithi196 ceph-mon[76349]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:49:12.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:13 smithi196 ceph-mon[76349]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:13 smithi184 ceph-mon[100622]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:13 smithi184 ceph-mon[102844]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:14.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:14.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:14.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:15 smithi184 ceph-mon[100622]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:15 smithi184 ceph-mon[102844]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:15 smithi196 ceph-mon[76349]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:17 smithi184 ceph-mon[100622]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:17 smithi184 ceph-mon[102844]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:17 smithi196 ceph-mon[76349]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:19 smithi196 ceph-mon[76349]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:19 smithi184 ceph-mon[100622]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:19 smithi184 ceph-mon[102844]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:20.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:20.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:20.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:21 smithi196 ceph-mon[76349]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:21 smithi184 ceph-mon[100622]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:21 smithi184 ceph-mon[102844]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:49:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:23 smithi184 ceph-mon[100622]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:23 smithi184 ceph-mon[102844]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:23 smithi196 ceph-mon[76349]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:24.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:24.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:24.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:24.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:24.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:25.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:25 smithi184 ceph-mon[100622]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:25.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:25 smithi184 ceph-mon[102844]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:25 smithi196 ceph-mon[76349]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:27 smithi196 ceph-mon[76349]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:27 smithi184 ceph-mon[100622]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:27 smithi184 ceph-mon[102844]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.363 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:49:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[102844]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[100622]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:29 smithi196 ceph-mon[76349]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:29.891 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:49:29.891 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T21:49:29.891 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T21:49:29.891 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:49:29.891 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (114m) 2s ago 119m 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (114m) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (115m) 2s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (113m) 2s ago 2h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (112m) 1s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (113m) 2s ago 2h 168M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 16.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (111m) 2s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (110m) 2s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (110m) 2s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (109m) 2s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (108m) 1s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (107m) 1s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (106m) 1s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:49:29.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (105m) 1s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:49:29.893 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (114m) 1s ago 2h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:49:29.893 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (105m) 2s ago 119m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:49:29.893 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (105m) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:49:29.893 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (105m) 2s ago 119m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:49:29.893 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (105m) 1s ago 119m 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:49:30.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:49:30.201 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:49:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:49:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:49:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:49:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:49:30.471 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:49:30.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:30 smithi184 ceph-mon[102844]: from='client.61925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:30.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:30 smithi184 ceph-mon[100622]: from='client.61925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:30 smithi196 ceph-mon[76349]: from='client.61925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:30.780 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:49:30.781 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:30.781 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:30.781 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:30.781 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:30.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:30.781 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:30.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:30.781 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:30.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:31.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:31 smithi196 ceph-mon[76349]: from='client.61931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:31 smithi196 ceph-mon[76349]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:31.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:31 smithi196 ceph-mon[76349]: from='client.61937 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1409153974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:31.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1993564290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[100622]: from='client.61931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[100622]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[100622]: from='client.61937 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1409153974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1993564290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[102844]: from='client.61931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[102844]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[102844]: from='client.61937 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1409153974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1993564290' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:32 smithi184 ceph-mon[100622]: from='client.51981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:32] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:49:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:32 smithi184 ceph-mon[102844]: from='client.51981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:32.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:32 smithi196 ceph-mon[76349]: from='client.51981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:32.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:33.364 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:33 smithi196 ceph-mon[76349]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:33.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:33 smithi184 ceph-mon[102844]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:33.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:33 smithi184 ceph-mon[100622]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:34.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:34.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:34.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:35 smithi196 ceph-mon[76349]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:35 smithi184 ceph-mon[100622]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:35 smithi184 ceph-mon[102844]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:37 smithi184 ceph-mon[100622]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:37 smithi184 ceph-mon[102844]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:37 smithi196 ceph-mon[76349]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:39 smithi184 ceph-mon[100622]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:39 smithi184 ceph-mon[102844]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:39 smithi196 ceph-mon[76349]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:40.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:40.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:40.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:41 smithi184 ceph-mon[100622]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:41 smithi184 ceph-mon[102844]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:41 smithi196 ceph-mon[76349]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:42] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T21:49:42.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:43 smithi184 ceph-mon[100622]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:43 smithi184 ceph-mon[102844]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:43 smithi196 ceph-mon[76349]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:44.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:44.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:44.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:44.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:44.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:45 smithi196 ceph-mon[76349]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:45 smithi184 ceph-mon[100622]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:45 smithi184 ceph-mon[102844]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:47 smithi184 ceph-mon[100622]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:47 smithi184 ceph-mon[102844]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:47 smithi196 ceph-mon[76349]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[100622]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[102844]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:49 smithi196 ceph-mon[76349]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:49:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:50.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:50.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:50.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:51 smithi184 ceph-mon[100622]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:51 smithi184 ceph-mon[102844]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:51.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:51 smithi196 ceph-mon[76349]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:52.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:49:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:49:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:53 smithi184 ceph-mon[100622]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:53 smithi184 ceph-mon[102844]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:53 smithi196 ceph-mon[76349]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:54.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:54.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:49:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:49:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:49:54.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:49:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:55 smithi196 ceph-mon[76349]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:55 smithi184 ceph-mon[100622]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:55 smithi184 ceph-mon[102844]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:57 smithi196 ceph-mon[76349]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:57 smithi184 ceph-mon[100622]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:57 smithi184 ceph-mon[102844]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:59 smithi196 ceph-mon[76349]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:59 smithi184 ceph-mon[102844]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:59 smithi184 ceph-mon[100622]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:49:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.595 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:00.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:00.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:00.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:01.078 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:50:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:01 smithi184 ceph-mon[100622]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:01 smithi184 ceph-mon[102844]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 12s ago 2h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (114m) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (114m) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:50:01.607 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (115m) 14s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (114m) 14s ago 2h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (113m) 12s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (113m) 14s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 16.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 12s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (112m) 14s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (111m) 14s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (110m) 14s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (110m) 14s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (108m) 12s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (108m) 12s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (106m) 12s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (106m) 12s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (115m) 12s ago 2h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (105m) 14s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:50:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (105m) 12s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:50:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (105m) 14s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:50:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (105m) 12s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:50:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:01 smithi196 ceph-mon[76349]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:01.917 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:50:01.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:50:01.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:50:01.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:01.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:50:01.918 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:50:02.189 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:50:02.190 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:50:02.542 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:02.542 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:02.542 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:02.542 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:02.542 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[100622]: from='client.61961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[100622]: from='client.61967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1213803822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[102844]: from='client.61961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[102844]: from='client.61967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1213803822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:02 smithi196 ceph-mon[76349]: from='client.61961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:02 smithi196 ceph-mon[76349]: from='client.61967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1213803822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:03 smithi196 ceph-mon[76349]: from='client.61973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:03 smithi196 ceph-mon[76349]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:03 smithi196 ceph-mon[76349]: from='client.61985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/610690755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[100622]: from='client.61973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[100622]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[100622]: from='client.61985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/610690755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[102844]: from='client.61973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[102844]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[102844]: from='client.61985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/610690755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:04.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:04.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:04.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:05.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:05 smithi184 ceph-mon[100622]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:05.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:05 smithi184 ceph-mon[102844]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:05 smithi196 ceph-mon[76349]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:07 smithi196 ceph-mon[76349]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:07 smithi184 ceph-mon[100622]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:07 smithi184 ceph-mon[102844]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[100622]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:08 smithi184 ceph-mon[102844]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:08 smithi196 ceph-mon[76349]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:11.027 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:10.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:11.027 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:10.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:11.027 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:10.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:11 smithi184 ceph-mon[100622]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:11 smithi184 ceph-mon[102844]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:11 smithi196 ceph-mon[76349]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:12.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:50:13.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:13 smithi196 ceph-mon[76349]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:13 smithi184 ceph-mon[100622]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:13 smithi184 ceph-mon[102844]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:14.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:14.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:14.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:15 smithi184 ceph-mon[100622]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:15 smithi184 ceph-mon[102844]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:15 smithi196 ceph-mon[76349]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:17 smithi184 ceph-mon[100622]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:17 smithi184 ceph-mon[102844]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:17 smithi196 ceph-mon[76349]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:19 smithi196 ceph-mon[76349]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:19 smithi184 ceph-mon[100622]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:19 smithi184 ceph-mon[102844]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:20.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:20.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:20.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:21 smithi196 ceph-mon[76349]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:21 smithi184 ceph-mon[100622]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:21 smithi184 ceph-mon[102844]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:50:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:23 smithi196 ceph-mon[76349]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:23 smithi184 ceph-mon[100622]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:23 smithi184 ceph-mon[102844]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:24.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:24.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:24.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:25 smithi184 ceph-mon[100622]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:25 smithi184 ceph-mon[102844]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:25 smithi196 ceph-mon[76349]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:27 smithi196 ceph-mon[76349]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:27 smithi184 ceph-mon[100622]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:27.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:27 smithi184 ceph-mon[102844]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[100622]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:28 smithi184 ceph-mon[102844]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:28 smithi196 ceph-mon[76349]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:30 smithi184 ceph-mon[100622]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:30 smithi184 ceph-mon[102844]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:30.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:30.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:30 smithi196 ceph-mon[76349]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:50:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:32 smithi184 ceph-mon[100622]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:32 smithi184 ceph-mon[102844]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:32.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:32 smithi196 ceph-mon[76349]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:32.843 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:50:33.367 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:50:33.367 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T21:50:33.367 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T21:50:33.367 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (115m) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (115m) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (116m) 5s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (114m) 5s ago 2h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (113m) 4s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (114m) 5s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (112m) 5s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (111m) 5s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (111m) 5s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (110m) 5s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (109m) 4s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (108m) 4s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:50:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (107m) 4s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (106m) 4s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (115m) 4s ago 2h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (106m) 5s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (106m) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (106m) 5s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:50:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (106m) 4s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:50:33.680 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:50:33.680 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:50:33.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:50:33.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:50:33.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:50:33.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[100622]: from='client.61997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[100622]: from='client.62003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[100622]: from='client.52041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[102844]: from='client.61997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[102844]: from='client.62003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:33 smithi184 ceph-mon[102844]: from='client.52041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:33 smithi196 ceph-mon[76349]: from='client.61997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:33 smithi196 ceph-mon[76349]: from='client.62003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:33 smithi196 ceph-mon[76349]: from='client.52041 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:50:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:50:33.952 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:50:34.260 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:34.260 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:34.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:34.260 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:34.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:34.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:34.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:34.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:34.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2544456440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:34.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:34 smithi196 ceph-mon[76349]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:34.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:34 smithi196 ceph-mon[76349]: from='client.62021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:34.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/97698751' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2544456440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[102844]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[102844]: from='client.62021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/97698751' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2544456440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[100622]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[100622]: from='client.62021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/97698751' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:36 smithi196 ceph-mon[76349]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:36 smithi184 ceph-mon[100622]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:36 smithi184 ceph-mon[102844]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:38 smithi196 ceph-mon[76349]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:38 smithi184 ceph-mon[100622]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:38 smithi184 ceph-mon[102844]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:40.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:40.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:40.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:40.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-mon[100622]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-mon[102844]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:40.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 3 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:40.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:40.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:40 smithi196 ceph-mon[76349]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:50:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:42 smithi196 ceph-mon[76349]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:42 smithi184 ceph-mon[100622]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:42 smithi184 ceph-mon[102844]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:44.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 5 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:44.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:44.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:44 smithi196 ceph-mon[76349]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:44 smithi184 ceph-mon[100622]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:44 smithi184 ceph-mon[102844]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:47.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:46 smithi184 ceph-mon[100622]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:47.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:46 smithi184 ceph-mon[102844]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:46 smithi196 ceph-mon[76349]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[100622]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:48 smithi184 ceph-mon[102844]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:48 smithi196 ceph-mon[76349]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:50:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:50.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:50.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:50.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:51 smithi184 ceph-mon[100622]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:51 smithi184 ceph-mon[102844]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:51 smithi196 ceph-mon[76349]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:52.296 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:50:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:50:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:53.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:53 smithi196 ceph-mon[76349]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:53 smithi184 ceph-mon[100622]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:53 smithi184 ceph-mon[102844]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:54.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:54.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:50:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:50:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:50:54.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:50:55.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:55 smithi196 ceph-mon[76349]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:55 smithi184 ceph-mon[100622]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:55 smithi184 ceph-mon[102844]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:57 smithi196 ceph-mon[76349]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:57 smithi184 ceph-mon[100622]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:57 smithi184 ceph-mon[102844]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:59 smithi196 ceph-mon[76349]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:59 smithi184 ceph-mon[100622]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:59 smithi184 ceph-mon[102844]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:00.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:00.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:00.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:01 smithi184 ceph-mon[100622]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:01 smithi184 ceph-mon[102844]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:01 smithi196 ceph-mon[76349]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:51:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:02.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:02.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:03 smithi196 ceph-mon[76349]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:51:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:03 smithi184 ceph-mon[100622]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:51:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:03 smithi184 ceph-mon[102844]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:51:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:04.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:04.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:04.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:04.559 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:51:05.086 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:51:05.086 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 15s ago 2h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (116m) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (115m) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (116m) 17s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (115m) 17s ago 2h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (114m) 15s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (114m) 17s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 15.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 15s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (113m) 17s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (112m) 17s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:51:05.087 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (111m) 17s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (111m) 17s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (110m) 15s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (109m) 15s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (108m) 15s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (107m) 15s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (116m) 15s ago 2h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (107m) 17s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (106m) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (106m) 17s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:51:05.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (106m) 15s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:51:05.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:05 smithi184 ceph-mon[100622]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:05.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:05 smithi184 ceph-mon[102844]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:51:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:51:05.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:51:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:05 smithi196 ceph-mon[76349]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:51:05.674 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:51:05.675 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:51:05.675 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:51:05.675 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:51:05.982 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:51:05.983 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:05.983 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:06.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[100622]: from='client.52065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[100622]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[100622]: from='client.52077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1162206454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:06.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2605016359' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:06.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[102844]: from='client.52065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[102844]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[102844]: from='client.52077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1162206454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:06.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2605016359' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:06 smithi196 ceph-mon[76349]: from='client.52065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:06 smithi196 ceph-mon[76349]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:06 smithi196 ceph-mon[76349]: from='client.52077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1162206454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2605016359' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:07.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:07 smithi184 ceph-mon[100622]: from='client.52089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:07.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:07 smithi184 ceph-mon[100622]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:07.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:07 smithi184 ceph-mon[102844]: from='client.52089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:07.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:07 smithi184 ceph-mon[102844]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:07 smithi196 ceph-mon[76349]: from='client.52089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:07 smithi196 ceph-mon[76349]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[100622]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[102844]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:08.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:08 smithi196 ceph-mon[76349]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:10.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:10.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:10.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:10.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:11 smithi184 ceph-mon[100622]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:11 smithi184 ceph-mon[102844]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:11 smithi196 ceph-mon[76349]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:51:12.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:13 smithi184 ceph-mon[100622]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:13 smithi184 ceph-mon[102844]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:13.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:13 smithi196 ceph-mon[76349]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:14.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:14.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:14.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:14.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:14 smithi196 ceph-mon[76349]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:14.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-mon[100622]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-mon[102844]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:16 smithi196 ceph-mon[76349]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:16 smithi184 ceph-mon[100622]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:16 smithi184 ceph-mon[102844]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:18 smithi196 ceph-mon[76349]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:18 smithi184 ceph-mon[100622]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:18 smithi184 ceph-mon[102844]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:20 smithi184 ceph-mon[100622]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:20 smithi184 ceph-mon[102844]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:20.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:20.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:20.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:20 smithi196 ceph-mon[76349]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:51:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:22 smithi196 ceph-mon[76349]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:22 smithi184 ceph-mon[102844]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:22 smithi184 ceph-mon[100622]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:24.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:24.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:24.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:25.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:24 smithi196 ceph-mon[76349]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:24 smithi184 ceph-mon[102844]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:24 smithi184 ceph-mon[100622]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:26 smithi196 ceph-mon[76349]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:27.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:26 smithi184 ceph-mon[100622]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:27.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:26 smithi184 ceph-mon[102844]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[100622]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[102844]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:28 smithi196 ceph-mon[76349]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[100622]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-mon[102844]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:30.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:30.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:30.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:30.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:30 smithi196 ceph-mon[76349]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:51:32.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:32 smithi184 ceph-mon[100622]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:32 smithi196 ceph-mon[76349]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:32 smithi184 ceph-mon[102844]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:34.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:34.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:34.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:34 smithi196 ceph-mon[76349]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:34 smithi184 ceph-mon[100622]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:34 smithi184 ceph-mon[102844]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:36.275 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:51:36.799 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:51:36.799 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (116m) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (116m) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (117m) 8s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (115m) 8s ago 2h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (115m) 7s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (115m) 8s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 15.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:51:36.800 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (113m) 8s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (113m) 8s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (112m) 8s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (111m) 8s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (110m) 7s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (109m) 7s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (108m) 7s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (107m) 7s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (116m) 7s ago 2h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (107m) 8s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (107m) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (107m) 8s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:51:36.801 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (107m) 7s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:51:37.111 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:51:37.111 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:51:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:51:37.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:51:37.113 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:51:37.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:36 smithi184 ceph-mon[100622]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:37.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:36 smithi184 ceph-mon[100622]: from='client.52101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:37.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:36 smithi184 ceph-mon[102844]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:37.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:36 smithi184 ceph-mon[102844]: from='client.52101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:36 smithi196 ceph-mon[76349]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:36 smithi196 ceph-mon[76349]: from='client.52101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:37.383 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:51:37.383 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:51:37.383 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:51:37.383 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:51:37.383 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:51:37.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:51:37.696 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:51:37.696 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:37.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:37.696 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:37.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[100622]: from='client.62075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[100622]: from='client.62081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1492590344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[100622]: from='client.62093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2353981494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[102844]: from='client.62075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[102844]: from='client.62081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1492590344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[102844]: from='client.62093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2353981494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:37 smithi196 ceph-mon[76349]: from='client.62075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:37 smithi196 ceph-mon[76349]: from='client.62081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1492590344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:37 smithi196 ceph-mon[76349]: from='client.62093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2353981494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:38 smithi196 ceph-mon[76349]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:38 smithi184 ceph-mon[100622]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:38 smithi184 ceph-mon[102844]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:40 smithi184 ceph-mon[100622]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:40 smithi184 ceph-mon[102844]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:40.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:40.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:40.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:40 smithi196 ceph-mon[76349]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:51:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:42 smithi196 ceph-mon[76349]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:42 smithi184 ceph-mon[100622]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:42 smithi184 ceph-mon[102844]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:44.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:44.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:44.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:44 smithi196 ceph-mon[76349]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:44 smithi184 ceph-mon[100622]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:44 smithi184 ceph-mon[102844]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:46 smithi196 ceph-mon[76349]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:46 smithi184 ceph-mon[100622]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:46 smithi184 ceph-mon[102844]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[100622]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[102844]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.123 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:49 smithi196 ceph-mon[76349]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:50.657 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:50.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:50.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:50.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:50.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:50.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[102844]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:50 smithi184 ceph-mon[100622]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:51:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:50 smithi196 ceph-mon[76349]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:51:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:51:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:52 smithi184 ceph-mon[100622]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:52 smithi184 ceph-mon[102844]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:52 smithi196 ceph-mon[76349]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:54.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:54.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:51:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:51:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:51:54.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:51:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:54 smithi196 ceph-mon[76349]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:54 smithi184 ceph-mon[100622]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:54 smithi184 ceph-mon[102844]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:56 smithi196 ceph-mon[76349]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:56 smithi184 ceph-mon[102844]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:56 smithi184 ceph-mon[100622]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:58 smithi196 ceph-mon[76349]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:51:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:58 smithi184 ceph-mon[100622]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:51:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:58 smithi184 ceph-mon[102844]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:51:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:00 smithi184 ceph-mon[100622]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:00 smithi184 ceph-mon[102844]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:00.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:01.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:00.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:01.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:00.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:00 smithi196 ceph-mon[76349]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:52:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:02 smithi196 ceph-mon[76349]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:02 smithi184 ceph-mon[100622]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:02 smithi184 ceph-mon[102844]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:04.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:04.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:04.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:04 smithi196 ceph-mon[76349]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:04 smithi184 ceph-mon[100622]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:04 smithi184 ceph-mon[102844]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:06 smithi196 ceph-mon[76349]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:06 smithi184 ceph-mon[100622]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:06 smithi184 ceph-mon[102844]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:07.995 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:52:08.539 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:52:08.539 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 2h - - 2024-09-20T21:52:08.539 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T21:52:08.539 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 0s ago 2h 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:52:08.539 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (117m) 0s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (116m) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (117m) 0s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (116m) 0s ago 2h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (115m) 18s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (115m) 0s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 0s ago 2h 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:52:08.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (114m) 0s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (113m) 0s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (112m) 0s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (112m) 0s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (111m) 18s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (110m) 18s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (109m) 18s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (108m) 18s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (117m) 18s ago 2h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (108m) 0s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:52:08.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (107m) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:52:08.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (107m) 0s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:52:08.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (107m) 18s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:52:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:52:08.860 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:52:09.131 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:52:09.131 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:52:09.131 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:52:09.131 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:52:09.131 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:52:09.132 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:52:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='client.62105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2402074447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:09.448 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:52:09.448 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:09.449 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:09.449 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:09.449 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:09.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:09.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='client.62105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2402074447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='client.62105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:09.450 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2402074447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='client.62117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='client.62129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1078009932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='client.62117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='client.62129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1078009932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:10.423 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='client.62117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='client.62129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1078009932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:10.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:10.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:10.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:10.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:10.787 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:10.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:11 smithi184 ceph-mon[100622]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:11 smithi184 ceph-mon[102844]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:11 smithi196 ceph-mon[76349]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:52:12.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:13 smithi196 ceph-mon[76349]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:13 smithi184 ceph-mon[100622]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:13 smithi184 ceph-mon[102844]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:14.034Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:14.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:14.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:15 smithi184 ceph-mon[100622]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:15 smithi184 ceph-mon[102844]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:15 smithi196 ceph-mon[76349]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:17.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:17 smithi196 ceph-mon[76349]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:17 smithi184 ceph-mon[100622]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:17 smithi184 ceph-mon[102844]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:19 smithi184 ceph-mon[100622]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:19 smithi184 ceph-mon[102844]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:19 smithi196 ceph-mon[76349]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:20.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:20.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:20.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:21 smithi184 ceph-mon[100622]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:21 smithi184 ceph-mon[102844]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:21 smithi196 ceph-mon[76349]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:52:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:23 smithi184 ceph-mon[100622]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:23 smithi184 ceph-mon[102844]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:23 smithi196 ceph-mon[76349]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:24.034Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:24.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:24.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:25 smithi184 ceph-mon[100622]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:25 smithi184 ceph-mon[102844]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:25.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:25 smithi196 ceph-mon[76349]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:27 smithi184 ceph-mon[100622]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:27 smithi184 ceph-mon[102844]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:27 smithi196 ceph-mon[76349]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:29.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[100622]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[102844]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:29 smithi196 ceph-mon[76349]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:30.762 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:30.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:30.763 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:30.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:30.763 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:30.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[100622]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[102844]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:30 smithi196 ceph-mon[76349]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:52:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:32 smithi184 ceph-mon[100622]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:32 smithi184 ceph-mon[102844]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:32 smithi196 ceph-mon[76349]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:34.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:34.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:34.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:34 smithi184 ceph-mon[100622]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:34 smithi184 ceph-mon[102844]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:34 smithi196 ceph-mon[76349]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:36.973 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:36 smithi196 ceph-mon[76349]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:36 smithi184 ceph-mon[100622]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:36 smithi184 ceph-mon[102844]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:38 smithi196 ceph-mon[76349]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:38 smithi184 ceph-mon[100622]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:38 smithi184 ceph-mon[102844]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:39.747 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:52:40.277 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:52:40.277 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T21:52:40.277 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T21:52:40.278 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:52:40.278 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:52:40.278 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (117m) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:52:40.278 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (117m) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (118m) 12s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (116m) 12s ago 2h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (116m) 10s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (116m) 12s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 15.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (114m) 12s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (114m) 12s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:52:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (113m) 12s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (112m) 12s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (111m) 10s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (110m) 10s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (109m) 10s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (108m) 10s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (117m) 10s ago 2h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (108m) 12s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (108m) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (108m) 12s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:52:40.281 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (108m) 10s ago 2h 90.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:52:40.588 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:52:40.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:52:40.589 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:52:40.856 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:52:40.857 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:52:40.857 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:52:40.857 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:52:40.857 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:52:40.857 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:52:40.857 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:40.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:40.857 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:40.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:40.857 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:40.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:41.169 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:52:41.169 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:41.169 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:41.169 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:41.169 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:41.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[102844]: from='client.62141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[102844]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[102844]: from='client.52179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[102844]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2863121411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[100622]: from='client.62141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[100622]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[100622]: from='client.52179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[100622]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2863121411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:40 smithi196 ceph-mon[76349]: from='client.62141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:40 smithi196 ceph-mon[76349]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:40 smithi196 ceph-mon[76349]: from='client.52179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:40 smithi196 ceph-mon[76349]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2863121411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[100622]: from='client.62165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2789416603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[102844]: from='client.62165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2789416603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:42.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:41 smithi196 ceph-mon[76349]: from='client.62165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:42.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2789416603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:42.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:52:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:42 smithi196 ceph-mon[76349]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:42 smithi184 ceph-mon[100622]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:42 smithi184 ceph-mon[102844]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:44.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:44.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:44.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:44 smithi184 ceph-mon[102844]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:44 smithi184 ceph-mon[100622]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:44 smithi196 ceph-mon[76349]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:46 smithi184 ceph-mon[100622]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:46 smithi184 ceph-mon[102844]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:46 smithi196 ceph-mon[76349]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[100622]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[102844]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:49 smithi196 ceph-mon[76349]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:50.989 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:50.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:50.989 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:50.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:50.989 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:50.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[100622]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[102844]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:51 smithi196 ceph-mon[76349]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:52:52.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:52.129 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:52:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:52:52.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:53 smithi184 ceph-mon[100622]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:53 smithi184 ceph-mon[102844]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:53 smithi196 ceph-mon[76349]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:54.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:54.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:52:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:52:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:52:54.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:52:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:55 smithi196 ceph-mon[76349]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:55 smithi184 ceph-mon[102844]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:55 smithi184 ceph-mon[100622]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:57 smithi196 ceph-mon[76349]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:57 smithi184 ceph-mon[100622]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:57 smithi184 ceph-mon[102844]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:59.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:59 smithi196 ceph-mon[76349]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:59.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:59 smithi184 ceph-mon[100622]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:59 smithi184 ceph-mon[102844]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:00.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:00.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:00.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:01 smithi196 ceph-mon[76349]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:01 smithi184 ceph-mon[100622]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:01 smithi184 ceph-mon[102844]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:53:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:03 smithi196 ceph-mon[76349]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:03.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:03 smithi184 ceph-mon[100622]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:03.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:03 smithi184 ceph-mon[102844]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:04.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:04.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:04.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:05 smithi196 ceph-mon[76349]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:05 smithi184 ceph-mon[100622]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:05 smithi184 ceph-mon[102844]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:07 smithi196 ceph-mon[76349]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:07 smithi184 ceph-mon[100622]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:07 smithi184 ceph-mon[102844]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:09 smithi196 ceph-mon[76349]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[102844]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[100622]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:09.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:10.881 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:10.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:10.882 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:10.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:10.882 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:10.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:11.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[100622]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:11.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[102844]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:11.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.471 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.471 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:53:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:11 smithi196 ceph-mon[76349]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 2h - - 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 3s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 91.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (118m) 3s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (117m) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:53:12.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (118m) 3s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (117m) 3s ago 2h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (116m) 1s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (116m) 3s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 3s ago 2h 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (115m) 3s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (114m) 3s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (113m) 3s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (113m) 3s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (112m) 1s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (111m) 1s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:53:12.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (110m) 1s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (109m) 1s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (118m) 1s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (109m) 3s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (108m) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (109m) 3s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:53:12.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (108m) 1s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:53:12.097 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:53:12.348 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:53:12.349 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:53:12.349 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:12.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:12.633 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:53:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:53:12.633 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:53:12.634 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:53:12.949 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:53:12.949 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:12.949 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:12.949 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:12.949 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: from='client.62177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: from='client.52215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: from='client.62189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4029539226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3549348557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: from='client.62177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: from='client.52215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: from='client.62189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4029539226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:13.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3549348557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: from='client.62177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: from='client.52215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: from='client.62189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4029539226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3549348557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-mon[102844]: from='client.52233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-mon[100622]: from='client.52233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:14.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:14.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:14.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:14.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:14 smithi196 ceph-mon[76349]: from='client.52233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:15 smithi184 ceph-mon[100622]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:15 smithi184 ceph-mon[102844]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:15 smithi196 ceph-mon[76349]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:17 smithi184 ceph-mon[100622]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:17 smithi184 ceph-mon[102844]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:17 smithi196 ceph-mon[76349]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:19.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:19 smithi196 ceph-mon[76349]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:19 smithi184 ceph-mon[100622]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:19 smithi184 ceph-mon[102844]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:20.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:20.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:20.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:21 smithi184 ceph-mon[100622]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:21 smithi184 ceph-mon[102844]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:21 smithi196 ceph-mon[76349]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T21:53:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:23 smithi184 ceph-mon[100622]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:23 smithi184 ceph-mon[102844]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:23 smithi196 ceph-mon[76349]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:24.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:24.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:24.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:25 smithi184 ceph-mon[100622]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:25 smithi184 ceph-mon[102844]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:25 smithi196 ceph-mon[76349]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:27 smithi184 ceph-mon[100622]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:27 smithi184 ceph-mon[102844]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:27 smithi196 ceph-mon[76349]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[102844]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[100622]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:29 smithi196 ceph-mon[76349]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:30.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:30.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:30.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[102844]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[100622]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:31 smithi196 ceph-mon[76349]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:53:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:32.664 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:32.664 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:33 smithi184 ceph-mon[100622]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:33 smithi184 ceph-mon[102844]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:33 smithi196 ceph-mon[76349]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:34.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:34.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:34.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:34.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:35 smithi196 ceph-mon[76349]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:35 smithi184 ceph-mon[100622]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:35 smithi184 ceph-mon[102844]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:37 smithi196 ceph-mon[76349]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:37 smithi184 ceph-mon[100622]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:37 smithi184 ceph-mon[102844]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:39 smithi196 ceph-mon[76349]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:39 smithi184 ceph-mon[100622]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:39 smithi184 ceph-mon[102844]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:40.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:40.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:40.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:41 smithi196 ceph-mon[76349]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:41 smithi184 ceph-mon[100622]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:41 smithi184 ceph-mon[102844]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:53:42.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:43.246 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:53:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:43 smithi184 ceph-mon[100622]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:43 smithi184 ceph-mon[102844]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:43 smithi196 ceph-mon[76349]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 2h - - 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 13s ago 2h 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (118m) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (118m) 13s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (119m) 15s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (117m) 15s ago 2h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (117m) 13s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:53:43.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (117m) 15s ago 2h 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 13s ago 2h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (116m) 15s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (115m) 15s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (114m) 15s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (113m) 15s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (112m) 13s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (111m) 13s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (110m) 13s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (110m) 13s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (118m) 13s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (109m) 15s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (109m) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (109m) 15s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:53:43.781 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (109m) 13s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:53:44.097 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:53:44.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:53:44.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:53:44.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:44.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:53:44.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:53:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:44.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:44.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:44.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:44.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:53:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:53:44.370 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:53:44.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:44 smithi196 ceph-mon[76349]: from='client.52245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/648265733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:53:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:44.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:44.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:44.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[100622]: from='client.52245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:44.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/648265733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:44.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[102844]: from='client.52245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:44.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/648265733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:45 smithi196 ceph-mon[76349]: from='client.62219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:45 smithi196 ceph-mon[76349]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:45 smithi196 ceph-mon[76349]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:45 smithi196 ceph-mon[76349]: from='client.62237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3664994605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[100622]: from='client.62219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[100622]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[100622]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[100622]: from='client.62237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3664994605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[102844]: from='client.62219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[102844]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[102844]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[102844]: from='client.62237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3664994605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:47 smithi196 ceph-mon[76349]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:47 smithi184 ceph-mon[100622]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:47 smithi184 ceph-mon[102844]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:49 smithi196 ceph-mon[76349]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[100622]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[102844]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:50.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:50.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:50.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:51.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:51 smithi196 ceph-mon[76349]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:51.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[100622]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[102844]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:53:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:53:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:53:52.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:52.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:53 smithi184 ceph-mon[100622]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:53 smithi184 ceph-mon[102844]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:53 smithi196 ceph-mon[76349]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:54.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:54.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:53:54.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:53:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:53:54.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:53:54.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:54 smithi184 ceph-mon[100622]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:54.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:54 smithi184 ceph-mon[102844]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:54 smithi196 ceph-mon[76349]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:56 smithi196 ceph-mon[76349]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:56 smithi184 ceph-mon[100622]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:56 smithi184 ceph-mon[102844]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:58 smithi196 ceph-mon[76349]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:53:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:58 smithi184 ceph-mon[100622]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:53:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:58 smithi184 ceph-mon[102844]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:53:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:00.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:00.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:00.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:00.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:00.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:00.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:01.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:00 smithi196 ceph-mon[76349]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:00 smithi184 ceph-mon[100622]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:00 smithi184 ceph-mon[102844]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:54:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:02 smithi196 ceph-mon[76349]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:02 smithi184 ceph-mon[100622]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:02 smithi184 ceph-mon[102844]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:04.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:04.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:04.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:04.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:04.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:04 smithi196 ceph-mon[76349]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:04 smithi184 ceph-mon[100622]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:04 smithi184 ceph-mon[102844]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:06 smithi196 ceph-mon[76349]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:54:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:06 smithi184 ceph-mon[100622]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:54:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:06 smithi184 ceph-mon[102844]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T21:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:08 smithi196 ceph-mon[76349]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:54:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:08 smithi184 ceph-mon[100622]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:54:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:08 smithi184 ceph-mon[102844]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T21:54:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:10.928 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:10.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:10.928 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:10.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:10.928 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:10.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[102844]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[100622]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:10 smithi196 ceph-mon[76349]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:12.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:54:13.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:12 smithi184 ceph-mon[100622]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:54:13.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:12 smithi184 ceph-mon[102844]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:54:13.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:12 smithi196 ceph-mon[76349]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:54:14.117 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.117 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.117 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:14.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:14.117 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:14.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:14.117 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:14.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.978 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:54:15.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:14 smithi184 ceph-mon[100622]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:54:15.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:14 smithi184 ceph-mon[102844]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:54:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:14 smithi196 ceph-mon[76349]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 2h - - 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 6s ago 2h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (119m) 6s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (119m) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (119m) 6s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (118m) 6s ago 2h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:54:15.511 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (117m) 4s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (117m) 6s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 6s ago 2h 15.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (116m) 6s ago 2h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (115m) 6s ago 2h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (114m) 6s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (114m) 6s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (113m) 4s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (112m) 4s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (111m) 4s ago 2h 72.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (110m) 4s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (119m) 4s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (110m) 6s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (109m) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:54:15.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (110m) 6s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:54:15.513 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (109m) 4s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:54:15.820 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:54:15.820 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:54:15.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:54:15.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:15.820 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:54:15.821 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:54:16.094 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:54:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:54:16.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[100622]: from='client.62249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[100622]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1066612540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[102844]: from='client.62249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[102844]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.096 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1066612540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:15 smithi196 ceph-mon[76349]: from='client.62249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:15 smithi196 ceph-mon[76349]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1066612540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:16.403 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:54:16.403 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:16.403 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:16.403 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:16.403 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[100622]: from='client.52293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[100622]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[100622]: from='client.62273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3384155185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[102844]: from='client.52293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[102844]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[102844]: from='client.62273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3384155185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:16 smithi196 ceph-mon[76349]: from='client.52293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:16 smithi196 ceph-mon[76349]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:54:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:16 smithi196 ceph-mon[76349]: from='client.62273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3384155185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:18 smithi184 ceph-mon[102844]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T21:54:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:18 smithi184 ceph-mon[100622]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T21:54:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:18 smithi196 ceph-mon[76349]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T21:54:20.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:20.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:20.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:20.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:20.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:20.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:21.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:20 smithi196 ceph-mon[76349]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:54:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:20 smithi184 ceph-mon[102844]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:54:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:20 smithi184 ceph-mon[100622]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T21:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:54:22.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:22 smithi196 ceph-mon[76349]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:54:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:22 smithi184 ceph-mon[102844]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:54:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:22 smithi184 ceph-mon[100622]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T21:54:24.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:24.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:24.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:24.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:24.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:24 smithi184 ceph-mon[100622]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:54:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:24 smithi184 ceph-mon[102844]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:54:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:24 smithi196 ceph-mon[76349]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T21:54:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:26 smithi184 ceph-mon[100622]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:54:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:26 smithi184 ceph-mon[102844]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:54:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:26 smithi196 ceph-mon[76349]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:54:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:28 smithi184 ceph-mon[102844]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T21:54:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:28 smithi184 ceph-mon[100622]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T21:54:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:28 smithi196 ceph-mon[76349]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-20T21:54:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:30.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:30.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:30.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:31.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[102844]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[100622]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:31 smithi196 ceph-mon[76349]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T21:54:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:54:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:33 smithi196 ceph-mon[76349]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:54:33.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:33 smithi184 ceph-mon[100622]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:54:33.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:33 smithi184 ceph-mon[102844]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:54:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:34.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:34.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:34.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:34.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:35.191 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:35 smithi184 ceph-mon[100622]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:54:35.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:35 smithi184 ceph-mon[102844]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:54:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:35 smithi196 ceph-mon[76349]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T21:54:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:37 smithi196 ceph-mon[76349]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T21:54:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:37 smithi184 ceph-mon[100622]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T21:54:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:37 smithi184 ceph-mon[102844]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T21:54:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:39 smithi196 ceph-mon[76349]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:54:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:39 smithi184 ceph-mon[100622]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:54:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:39 smithi184 ceph-mon[102844]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:54:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:40.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:40.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:40.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:41.371 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:41 smithi184 ceph-mon[100622]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T21:54:41.371 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:41 smithi184 ceph-mon[102844]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T21:54:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:41 smithi196 ceph-mon[76349]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T21:54:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:54:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:43.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:43 smithi196 ceph-mon[76349]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:54:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:43 smithi184 ceph-mon[100622]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:54:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:43 smithi184 ceph-mon[102844]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T21:54:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:44.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:44.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:44.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:45 smithi196 ceph-mon[76349]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:45 smithi184 ceph-mon[100622]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:45 smithi184 ceph-mon[102844]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:46.693 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 2h - - 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 2h - - 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 16s ago 2h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (119m) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (119m) 16s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:54:47.226 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (118m) 18s ago 2h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (118m) 16s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (118m) 18s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 15.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 16s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (117m) 18s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (116m) 18s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (115m) 18s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (115m) 18s ago 2h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (113m) 16s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (112m) 16s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (111m) 16s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (111m) 16s ago 2h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (119m) 16s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:54:47.227 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (110m) 18s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:54:47.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (110m) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:54:47.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (110m) 18s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:54:47.228 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (110m) 16s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:54:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:47 smithi196 ceph-mon[76349]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:47.539 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:54:47.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:54:47.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:54:47.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:47.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:54:47.540 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:54:47.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:47 smithi184 ceph-mon[100622]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:47.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:47 smithi184 ceph-mon[102844]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:47.808 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:54:47.809 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:54:48.123 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:54:48.123 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:48.123 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:48 smithi196 ceph-mon[76349]: from='client.62285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:48 smithi196 ceph-mon[76349]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:48 smithi196 ceph-mon[76349]: from='client.62297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/736653815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[100622]: from='client.62285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[100622]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[100622]: from='client.62297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/736653815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[102844]: from='client.62285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[102844]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[102844]: from='client.62297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/736653815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:49.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:49 smithi196 ceph-mon[76349]: from='client.52341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:49.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:49 smithi196 ceph-mon[76349]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:49.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2642262365' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[102844]: from='client.52341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[102844]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2642262365' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[100622]: from='client.52341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[100622]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:49.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2642262365' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:50.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:50.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:50.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[100622]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T21:54:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[102844]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:51 smithi196 ceph-mon[76349]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T21:54:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:54:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:54:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:54:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:53 smithi184 ceph-mon[100622]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:53 smithi184 ceph-mon[102844]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:53 smithi196 ceph-mon[76349]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:54:54.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:54.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:54.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:54.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:54:54.216 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:54:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:54:54.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:54:55.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:55 smithi184 ceph-mon[100622]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:55.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:55 smithi184 ceph-mon[102844]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:55 smithi196 ceph-mon[76349]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:54:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:57 smithi184 ceph-mon[100622]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:57 smithi184 ceph-mon[102844]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:57 smithi196 ceph-mon[76349]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:59 smithi184 ceph-mon[100622]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:54:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:59 smithi184 ceph-mon[102844]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:54:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:59 smithi196 ceph-mon[76349]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:54:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:54:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:00.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:00.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:00.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:01 smithi196 ceph-mon[76349]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:55:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:01 smithi184 ceph-mon[100622]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:55:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:01 smithi184 ceph-mon[102844]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:55:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:55:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:03.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:03 smithi184 ceph-mon[100622]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:03.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:03 smithi184 ceph-mon[102844]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:03 smithi196 ceph-mon[76349]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:04.491 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:04.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:04.492 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:04.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:04.492 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:04.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:05 smithi196 ceph-mon[76349]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:05 smithi184 ceph-mon[100622]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:05 smithi184 ceph-mon[102844]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:07 smithi184 ceph-mon[100622]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:07 smithi184 ceph-mon[102844]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:07 smithi196 ceph-mon[76349]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:09 smithi184 ceph-mon[100622]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:09 smithi184 ceph-mon[102844]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:09 smithi196 ceph-mon[76349]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:10.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:10.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:10.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:10.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:11 smithi196 ceph-mon[76349]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[100622]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[102844]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:55:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:13 smithi196 ceph-mon[76349]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:13 smithi184 ceph-mon[100622]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:13 smithi184 ceph-mon[102844]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:14.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:14.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:14.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:14.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:14.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:14.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:14.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:15.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:15 smithi184 ceph-mon[100622]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:15.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:15 smithi184 ceph-mon[102844]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:15 smithi196 ceph-mon[76349]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:17.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:17 smithi184 ceph-mon[100622]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:17.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:17 smithi184 ceph-mon[102844]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:17 smithi196 ceph-mon[76349]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:18.408 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:55:18.934 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:55:18.934 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 2h - - 2024-09-20T21:55:18.934 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T21:55:18.934 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 9s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:55:18.934 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 9s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 9s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (119m) 9s ago 2h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (118m) 7s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (119m) 9s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 9s ago 2h 16.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (117m) 9s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (116m) 9s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (116m) 9s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (115m) 9s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (114m) 7s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:55:18.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (113m) 7s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (112m) 7s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (111m) 7s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (111m) 9s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (111m) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (111m) 9s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:55:18.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (110m) 7s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:55:19.247 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:55:19.247 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:55:19.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:55:19.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:55:19.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:55:19.520 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:55:19.521 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:55:19.521 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:55:19.521 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:55:19.521 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[100622]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[100622]: from='client.62321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2938133051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[102844]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[102844]: from='client.62321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:19.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2938133051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:19 smithi196 ceph-mon[76349]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:19 smithi196 ceph-mon[76349]: from='client.62321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2938133051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:19.832 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:55:19.832 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:19.832 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:19.832 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:19.833 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:20.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[100622]: from='client.52359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[100622]: from='client.62333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3172337941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:20.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[102844]: from='client.52359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[102844]: from='client.62333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3172337941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:20 smithi196 ceph-mon[76349]: from='client.52359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:20 smithi196 ceph-mon[76349]: from='client.62333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3172337941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:20.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:20.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:20.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:21 smithi196 ceph-mon[76349]: from='client.52377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:21 smithi196 ceph-mon[76349]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:21 smithi184 ceph-mon[100622]: from='client.52377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:21 smithi184 ceph-mon[100622]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:21 smithi184 ceph-mon[102844]: from='client.52377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:21 smithi184 ceph-mon[102844]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:55:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:23 smithi196 ceph-mon[76349]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:23 smithi184 ceph-mon[100622]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:23 smithi184 ceph-mon[102844]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:24.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:24.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:24.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:25 smithi196 ceph-mon[76349]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:25 smithi184 ceph-mon[100622]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:25 smithi184 ceph-mon[102844]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:27 smithi196 ceph-mon[76349]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:27 smithi184 ceph-mon[100622]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:27 smithi184 ceph-mon[102844]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:29 smithi196 ceph-mon[76349]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:29 smithi184 ceph-mon[100622]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:29 smithi184 ceph-mon[102844]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:30.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:30.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:30.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:30.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[100622]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[102844]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:31 smithi196 ceph-mon[76349]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:55:32.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:32.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:33 smithi184 ceph-mon[102844]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:33 smithi184 ceph-mon[100622]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:33 smithi196 ceph-mon[76349]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:34.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:34.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:34.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:34.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:34.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:34.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:34 smithi184 ceph-mon[100622]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:34 smithi184 ceph-mon[102844]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:34 smithi196 ceph-mon[76349]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:37.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:36 smithi184 ceph-mon[100622]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:37.125 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:36 smithi184 ceph-mon[102844]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:36 smithi196 ceph-mon[76349]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:38 smithi184 ceph-mon[100622]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:38 smithi184 ceph-mon[102844]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:38 smithi196 ceph-mon[76349]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:40.915 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:40.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:40.915 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:40.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:40.915 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:40.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:40 smithi184 ceph-mon[100622]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:40 smithi184 ceph-mon[102844]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:40 smithi196 ceph-mon[76349]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:42.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:55:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:42 smithi196 ceph-mon[76349]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:42 smithi184 ceph-mon[100622]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:42 smithi184 ceph-mon[102844]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:44.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:44.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:44.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:44.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:45.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:44 smithi196 ceph-mon[76349]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:44 smithi184 ceph-mon[100622]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:44 smithi184 ceph-mon[102844]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:46 smithi184 ceph-mon[100622]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:46 smithi184 ceph-mon[102844]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:46 smithi196 ceph-mon[76349]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:48 smithi184 ceph-mon[100622]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:48 smithi184 ceph-mon[102844]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:48 smithi196 ceph-mon[76349]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:50.123 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:55:50.650 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:55:50.650 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T21:55:50.650 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 2h - - 2024-09-20T21:55:50.650 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 19s ago 2h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (119m) 1s ago 2h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (119m) 19s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (119m) 1s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 19s ago 2h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (118m) 1s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (117m) 1s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (116m) 1s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (116m) 1s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (114m) 19s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:55:50.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (113m) 19s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (112m) 19s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (112m) 19s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (111m) 1s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (111m) 19s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (111m) 1s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:55:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (111m) 19s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: from='client.62357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:50 smithi196 ceph-mon[76349]: from='client.62363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.958 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:55:50.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:55:50.967 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:55:50.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: from='client.62357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[100622]: from='client.62363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: from='client.62357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-mon[102844]: from='client.62363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:50.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:50.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:50.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:50.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:50.969 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:50.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:55:51.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:55:51.232 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:55:51.542 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:55:51.542 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:51.543 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:51.543 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:51.543 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='client.62369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3614289676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='client.52413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2250490516' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='client.62369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3614289676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='client.52413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:51.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2250490516' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='client.62369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3614289676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:55:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='client.52413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:51.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2250490516' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:55:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:55:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:52 smithi196 ceph-mon[76349]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:52 smithi184 ceph-mon[100622]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:52 smithi184 ceph-mon[102844]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:54.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:54.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:55:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:55:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:55:54.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:55:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:54 smithi184 ceph-mon[100622]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:54 smithi184 ceph-mon[102844]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:54 smithi196 ceph-mon[76349]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:57.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:56 smithi184 ceph-mon[100622]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:57.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:56 smithi184 ceph-mon[102844]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:56 smithi196 ceph-mon[76349]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:58 smithi184 ceph-mon[100622]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:55:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:58 smithi184 ceph-mon[102844]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:55:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:58 smithi196 ceph-mon[76349]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:55:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:00.923 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:00.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:00.924 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:00.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:00.924 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:00.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:00 smithi184 ceph-mon[100622]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:00 smithi184 ceph-mon[102844]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:00 smithi196 ceph-mon[76349]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:56:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:03.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:02 smithi196 ceph-mon[76349]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:02 smithi184 ceph-mon[100622]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:02 smithi184 ceph-mon[102844]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:04.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:04.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:04.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:04 smithi184 ceph-mon[100622]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:04 smithi184 ceph-mon[102844]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:04 smithi196 ceph-mon[76349]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:06 smithi184 ceph-mon[100622]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:06 smithi184 ceph-mon[102844]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:06 smithi196 ceph-mon[76349]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:08 smithi184 ceph-mon[100622]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:08 smithi184 ceph-mon[102844]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:08 smithi196 ceph-mon[76349]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[100622]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-mon[102844]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:11.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:10.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:11.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:10.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:11.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:10.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:10 smithi196 ceph-mon[76349]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:56:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:13 smithi196 ceph-mon[76349]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:13 smithi184 ceph-mon[100622]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:13 smithi184 ceph-mon[102844]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:14.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:14.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:14.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:14.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:14.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:14.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:15 smithi196 ceph-mon[76349]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:15 smithi184 ceph-mon[100622]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:15 smithi184 ceph-mon[102844]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:17 smithi184 ceph-mon[100622]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:17 smithi184 ceph-mon[102844]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:17 smithi196 ceph-mon[76349]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:19.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:19 smithi196 ceph-mon[76349]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:19 smithi184 ceph-mon[100622]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:19 smithi184 ceph-mon[102844]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:20.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:20.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:20.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:21 smithi196 ceph-mon[76349]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:21 smithi184 ceph-mon[100622]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:21 smithi184 ceph-mon[102844]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:21.841 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 2h - - 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 28.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 12s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:56:22.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 2h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (119m) 11s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (118m) 12s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (117m) 12s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (117m) 12s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (116m) 12s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (115m) 11s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (114m) 11s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (113m) 11s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (112m) 11s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:56:22.370 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:56:22.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (112m) 12s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:56:22.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (112m) 11s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:56:22.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (112m) 12s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:56:22.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (111m) 11s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:56:22.676 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:56:22.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:56:22.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:56:22.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:56:22.677 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:56:22.944 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:56:22.944 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:56:22.944 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:56:22.944 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:56:22.945 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:56:23.259 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:56:23.259 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:23.259 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:23.259 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:23.259 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.52425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.62399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.62405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4155785156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2013284611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.52425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.62399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.62405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4155785156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:23.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2013284611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.52425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.62399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.62405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4155785156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2013284611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:24.366 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:24.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:24.366 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:24.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:24.366 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:24.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:24 smithi196 ceph-mon[76349]: from='client.52449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:24 smithi184 ceph-mon[102844]: from='client.52449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:24 smithi184 ceph-mon[100622]: from='client.52449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:25.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:25 smithi196 ceph-mon[76349]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:25 smithi184 ceph-mon[102844]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:25 smithi184 ceph-mon[100622]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:27 smithi196 ceph-mon[76349]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:27 smithi184 ceph-mon[102844]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:27 smithi184 ceph-mon[100622]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:29 smithi196 ceph-mon[76349]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:29 smithi184 ceph-mon[102844]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:29 smithi184 ceph-mon[100622]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[100622]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-mon[102844]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:30.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:30.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:30.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:30 smithi196 ceph-mon[76349]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:56:32.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:33 smithi184 ceph-mon[100622]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:33 smithi184 ceph-mon[102844]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:33 smithi196 ceph-mon[76349]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:34.511 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:34.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:34.511 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:34.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:34.512 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:34.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:34 smithi184 ceph-mon[102844]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:34 smithi184 ceph-mon[100622]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:34 smithi196 ceph-mon[76349]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:37.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:36 smithi184 ceph-mon[100622]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:37.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:36 smithi184 ceph-mon[102844]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:36 smithi196 ceph-mon[76349]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:39.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:38 smithi184 ceph-mon[100622]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:39.152 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:38 smithi184 ceph-mon[102844]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:38 smithi196 ceph-mon[76349]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:40.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:40.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:40.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:40.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:40.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:40.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:40 smithi184 ceph-mon[100622]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:40 smithi184 ceph-mon[102844]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:40 smithi196 ceph-mon[76349]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:56:43.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:43.193 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:42 smithi196 ceph-mon[76349]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:42 smithi184 ceph-mon[102844]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:42 smithi184 ceph-mon[100622]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:44.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:44.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:44.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:44.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:45.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:44 smithi196 ceph-mon[76349]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:44 smithi184 ceph-mon[100622]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:44 smithi184 ceph-mon[102844]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:47.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:46 smithi196 ceph-mon[76349]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:46 smithi184 ceph-mon[100622]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:46 smithi184 ceph-mon[102844]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:49.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:48 smithi184 ceph-mon[102844]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:49.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:48 smithi184 ceph-mon[100622]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:48 smithi196 ceph-mon[76349]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:50.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:50.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:50.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:50.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:50.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:50.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[102844]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[100622]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:50 smithi196 ceph-mon[76349]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:56:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T21:56:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:52 smithi196 ceph-mon[76349]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[102844]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:56:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:52 smithi184 ceph-mon[100622]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 2h - - 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 2h - - 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 29.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 2s ago 2h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:56:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 2s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 2h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 2s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (119m) 4s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (118m) 4s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (117m) 4s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (117m) 4s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:56:54.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (115m) 2s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (114m) 2s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (113m) 2s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (113m) 2s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 2s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (112m) 4s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (112m) 2s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (112m) 4s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:56:54.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (112m) 2s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:56:54.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:56:54.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:56:54.394 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:56:54.394 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:56:54.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:56:54.394 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:56:54.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:54.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:54.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:54.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:56:54.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:56:54.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:56:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:56:54.663 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:56:54.663 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:56:54.663 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:56:54.663 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:56:54.974 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:56:54.974 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:54.974 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:54.974 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[100622]: from='client.62429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[100622]: from='client.52467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[100622]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[100622]: from='client.52473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3031293247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[102844]: from='client.62429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[102844]: from='client.52467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[102844]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[102844]: from='client.52473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3031293247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:54 smithi196 ceph-mon[76349]: from='client.62429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:54 smithi196 ceph-mon[76349]: from='client.52467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:54 smithi196 ceph-mon[76349]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:54 smithi196 ceph-mon[76349]: from='client.52473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3031293247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:55 smithi184 ceph-mon[100622]: from='client.62453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1614595193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:55 smithi184 ceph-mon[102844]: from='client.62453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1614595193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:55 smithi196 ceph-mon[76349]: from='client.62453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1614595193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:56 smithi184 ceph-mon[100622]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:56 smithi184 ceph-mon[102844]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:56 smithi196 ceph-mon[76349]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:58 smithi184 ceph-mon[100622]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:56:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:58 smithi184 ceph-mon[102844]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:56:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:58 smithi196 ceph-mon[76349]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:56:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:00.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:00.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:00.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:00.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:00.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:00.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:00 smithi184 ceph-mon[100622]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:00 smithi184 ceph-mon[102844]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:00 smithi196 ceph-mon[76349]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:57:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:02 smithi184 ceph-mon[100622]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:02 smithi184 ceph-mon[102844]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:02 smithi196 ceph-mon[76349]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:04.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:04.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:04.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:05.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:04 smithi196 ceph-mon[76349]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:04 smithi184 ceph-mon[100622]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:04 smithi184 ceph-mon[102844]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:06.998 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:06 smithi196 ceph-mon[76349]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:06 smithi184 ceph-mon[100622]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:06 smithi184 ceph-mon[102844]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:08 smithi184 ceph-mon[100622]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:08 smithi184 ceph-mon[102844]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:08 smithi196 ceph-mon[76349]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:10.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:10.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:10.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:11 smithi196 ceph-mon[76349]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[100622]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:11.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[102844]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:11.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:11.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:57:12.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[100622]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:12.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:12 smithi184 ceph-mon[102844]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:13.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:13.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:13.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:13.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:13.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:12 smithi196 ceph-mon[76349]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:14.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:14.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:14.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:14.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:14 smithi196 ceph-mon[76349]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:14 smithi184 ceph-mon[100622]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:14 smithi184 ceph-mon[102844]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:16 smithi184 ceph-mon[100622]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:16 smithi184 ceph-mon[102844]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:16 smithi196 ceph-mon[76349]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:19.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:18 smithi184 ceph-mon[100622]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:19.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:18 smithi184 ceph-mon[102844]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:18 smithi196 ceph-mon[76349]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:20.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:20.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:20.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:20.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:20.947 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:20.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:21.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:20 smithi184 ceph-mon[100622]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:21.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:20 smithi184 ceph-mon[102844]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:20 smithi196 ceph-mon[76349]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:57:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:22 smithi184 ceph-mon[102844]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:22 smithi184 ceph-mon[100622]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:22 smithi196 ceph-mon[76349]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:24.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:24.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:24.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:24.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:25.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:24 smithi184 ceph-mon[100622]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:25.279 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:24 smithi184 ceph-mon[102844]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:25.279 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:57:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:24 smithi196 ceph-mon[76349]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:25.806 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:57:25.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T21:57:25.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T21:57:25.806 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 15s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 15s ago 2h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (119m) 15s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (118m) 15s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (118m) 15s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (117m) 15s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (116m) 14s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (115m) 14s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:57:25.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (114m) 14s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (113m) 14s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (113m) 15s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (113m) 14s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (113m) 15s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:57:25.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (113m) 14s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:57:26.122 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:57:26.122 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:25 smithi184 ceph-mon[100622]: from='client.52497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:26.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:25 smithi184 ceph-mon[102844]: from='client.52497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:26.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:25 smithi196 ceph-mon[76349]: from='client.52497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:57:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:57:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:57:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:57:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:57:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:57:26.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:57:26.699 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:57:26.699 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:26.699 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:26.699 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:26.699 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: from='client.52503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: from='client.62477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3015750283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: from='client.62489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4168962372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: from='client.52503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: from='client.62477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3015750283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: from='client.62489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4168962372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: from='client.52503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: from='client.62477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3015750283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: from='client.62489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4168962372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:28 smithi184 ceph-mon[100622]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:28 smithi184 ceph-mon[102844]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:28 smithi196 ceph-mon[76349]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:30.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:30.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:30.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[100622]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[102844]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:31.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:31 smithi196 ceph-mon[76349]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:32 smithi196 ceph-mon[76349]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:33.351 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.351 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.351 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.351 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[100622]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:33.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:33.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:33.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:32 smithi184 ceph-mon[102844]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:34.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:34.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:34.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:34.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:34 smithi184 ceph-mon[100622]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:34 smithi184 ceph-mon[102844]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:34 smithi196 ceph-mon[76349]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:36 smithi184 ceph-mon[100622]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:36 smithi184 ceph-mon[102844]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:36 smithi196 ceph-mon[76349]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:38 smithi184 ceph-mon[100622]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:38 smithi184 ceph-mon[102844]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:38 smithi196 ceph-mon[76349]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:40.856 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:40.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:40.856 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:40.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:40.856 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:40.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:41.116 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:40 smithi184 ceph-mon[100622]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:41.117 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:40 smithi184 ceph-mon[102844]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:40 smithi196 ceph-mon[76349]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:42.095 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T21:57:43.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:43.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:42 smithi196 ceph-mon[76349]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:42 smithi184 ceph-mon[102844]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:42 smithi184 ceph-mon[100622]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:44.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:44.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:44.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:44.375 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:44 smithi184 ceph-mon[100622]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:44 smithi184 ceph-mon[102844]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:44 smithi196 ceph-mon[76349]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:46 smithi196 ceph-mon[76349]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:46 smithi184 ceph-mon[100622]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:46 smithi184 ceph-mon[102844]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:48 smithi196 ceph-mon[76349]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:48 smithi184 ceph-mon[100622]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:48 smithi184 ceph-mon[102844]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:50.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:50.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:50.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:51 smithi196 ceph-mon[76349]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[100622]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[102844]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:57:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:57:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[100622]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[102844]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:52 smithi196 ceph-mon[76349]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:57:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:54.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:54.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:57:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:57:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:57:54.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:57:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:54 smithi184 ceph-mon[100622]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:54 smithi184 ceph-mon[102844]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:54 smithi196 ceph-mon[76349]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:56.995 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:57:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:56 smithi184 ceph-mon[100622]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:56 smithi184 ceph-mon[102844]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:56 smithi196 ceph-mon[76349]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:57.523 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:57:57.523 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T21:57:57.523 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 90.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 2h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (119m) 7s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:57:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (118m) 7s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (118m) 7s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (116m) 5s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (116m) 5s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (114m) 5s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (114m) 5s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (113m) 7s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (113m) 5s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (113m) 7s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:57:57.525 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (113m) 5s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:57:57.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:57:57.835 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:57:58.105 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:57:58.106 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[100622]: from='client.62501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[100622]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2745187319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[102844]: from='client.62501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[102844]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2745187319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:57 smithi196 ceph-mon[76349]: from='client.62501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:57 smithi196 ceph-mon[76349]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2745187319' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:58.416 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:57:58.416 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:58.416 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:58.416 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:58.416 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[100622]: from='client.52545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[100622]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[100622]: from='client.52557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/54301650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[102844]: from='client.52545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[102844]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[102844]: from='client.52557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/54301650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:58 smithi196 ceph-mon[76349]: from='client.52545 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:58 smithi196 ceph-mon[76349]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:58 smithi196 ceph-mon[76349]: from='client.52557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/54301650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:00.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:00.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:00.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:01.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:00 smithi184 ceph-mon[100622]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:01.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:00 smithi184 ceph-mon[102844]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:00 smithi196 ceph-mon[76349]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:02.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:58:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:02 smithi184 ceph-mon[102844]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:02 smithi184 ceph-mon[100622]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:02 smithi196 ceph-mon[76349]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:04.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:04.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:04.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:04 smithi184 ceph-mon[100622]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:04 smithi184 ceph-mon[102844]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:04 smithi196 ceph-mon[76349]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:07.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:06 smithi196 ceph-mon[76349]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:06 smithi184 ceph-mon[100622]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:06 smithi184 ceph-mon[102844]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:08 smithi184 ceph-mon[100622]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:08 smithi184 ceph-mon[102844]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:08 smithi196 ceph-mon[76349]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:10.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:10.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:10.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:10.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:10.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[100622]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[102844]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:11 smithi196 ceph-mon[76349]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:58:13.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[100622]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[102844]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-mon[76349]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:14.104 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 21:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T21:58:13.812872805Z level=info msg="Completed cleanup jobs" duration=12.767096ms 2024-09-20T21:58:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:14.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:14.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:14.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:15 smithi196 ceph-mon[76349]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:15 smithi184 ceph-mon[102844]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:15 smithi184 ceph-mon[100622]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:17 smithi196 ceph-mon[76349]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:17 smithi184 ceph-mon[100622]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:17 smithi184 ceph-mon[102844]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:19 smithi196 ceph-mon[76349]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:19 smithi184 ceph-mon[100622]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:19 smithi184 ceph-mon[102844]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:20.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:20.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:20.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:20.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:20.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:20.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:21.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:21 smithi184 ceph-mon[100622]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:21.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:21 smithi184 ceph-mon[102844]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:21 smithi196 ceph-mon[76349]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:22.213 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:58:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:23 smithi196 ceph-mon[76349]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:23.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:23.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:23 smithi184 ceph-mon[100622]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:23.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:23.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:23 smithi184 ceph-mon[102844]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:24.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:24.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:24.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:24.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:24.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:25 smithi196 ceph-mon[76349]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:25 smithi184 ceph-mon[100622]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:25 smithi184 ceph-mon[102844]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:27 smithi196 ceph-mon[76349]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:27 smithi184 ceph-mon[100622]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:27 smithi184 ceph-mon[102844]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:28.712 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:58:29.240 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:58:29.240 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 2h - - 2024-09-20T21:58:29.240 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T21:58:29.240 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 18s ago 2h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 18s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (119m) 18s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (119m) 18s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (118m) 18s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (117m) 17s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:58:29.241 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (116m) 17s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (115m) 17s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (114m) 17s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (114m) 18s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (114m) 17s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (114m) 18s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:58:29.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (114m) 17s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:58:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:29 smithi196 ceph-mon[76349]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:29 smithi184 ceph-mon[100622]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:29 smithi184 ceph-mon[102844]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:58:29.549 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:58:29.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:58:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:58:29.818 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:58:29.818 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:58:29.818 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:58:29.818 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:58:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:58:30.125 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:58:30.125 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:30.125 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:30 smithi196 ceph-mon[76349]: from='client.52569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:30 smithi196 ceph-mon[76349]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:30 smithi196 ceph-mon[76349]: from='client.62549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2349708506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4014150067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[100622]: from='client.52569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[100622]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[100622]: from='client.62549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2349708506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4014150067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[102844]: from='client.52569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[102844]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[102844]: from='client.62549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2349708506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4014150067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:30.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:30.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:30.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:31.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:31 smithi196 ceph-mon[76349]: from='client.52593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:31.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:31 smithi196 ceph-mon[76349]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:31.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[102844]: from='client.52593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[102844]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[100622]: from='client.52593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[100622]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:58:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[102844]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[100622]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:33 smithi196 ceph-mon[76349]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:34.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:34.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:34.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:34.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:34.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:35 smithi184 ceph-mon[102844]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:35 smithi184 ceph-mon[100622]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:35 smithi196 ceph-mon[76349]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:37 smithi184 ceph-mon[100622]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:37 smithi184 ceph-mon[102844]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:37 smithi196 ceph-mon[76349]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:39 smithi184 ceph-mon[100622]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:39 smithi184 ceph-mon[102844]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:39 smithi196 ceph-mon[76349]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:40.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:40.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:40.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:41 smithi184 ceph-mon[100622]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:41 smithi184 ceph-mon[102844]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:41.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:41 smithi196 ceph-mon[76349]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:58:43.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:43.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:43 smithi184 ceph-mon[102844]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:43.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:43.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:43 smithi184 ceph-mon[100622]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:43 smithi196 ceph-mon[76349]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:44.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:44.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:44.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:45 smithi184 ceph-mon[100622]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:45 smithi184 ceph-mon[102844]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:45 smithi196 ceph-mon[76349]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:47 smithi184 ceph-mon[100622]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:47 smithi184 ceph-mon[102844]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:47 smithi196 ceph-mon[76349]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:49 smithi184 ceph-mon[100622]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:49 smithi184 ceph-mon[102844]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:49.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:49 smithi196 ceph-mon[76349]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:50.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:50.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:50.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:51 smithi196 ceph-mon[76349]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[102844]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[100622]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:51.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:58:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:58:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:53 smithi196 ceph-mon[76349]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[100622]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[102844]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:58:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:54.065Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:54.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:58:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:58:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:58:54.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:58:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:55 smithi196 ceph-mon[76349]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:55 smithi184 ceph-mon[100622]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:55 smithi184 ceph-mon[102844]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:57 smithi196 ceph-mon[76349]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:57 smithi184 ceph-mon[100622]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:57 smithi184 ceph-mon[102844]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:59 smithi196 ceph-mon[76349]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:59.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:59 smithi184 ceph-mon[100622]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:59 smithi184 ceph-mon[102844]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:00.423 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:59:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:00.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:00.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:00.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 2h - - 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 2h - - 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 29.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 8s ago 2h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 2h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 8s ago 2h 168M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:59:00.949 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 8s ago 2h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (119m) 10s ago 2h 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (119m) 10s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (117m) 8s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (117m) 8s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (115m) 8s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (115m) 8s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 8s ago 2h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (114m) 10s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:59:00.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (114m) 8s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:59:00.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (114m) 10s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:59:00.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (114m) 8s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:59:01.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:59:01.259 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:59:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:59:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:59:01.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[100622]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:01.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[100622]: from='client.52605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:01.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3760767076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:01.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[102844]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:01.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[102844]: from='client.52605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:01.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3760767076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:01 smithi196 ceph-mon[76349]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:01 smithi196 ceph-mon[76349]: from='client.52605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3760767076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:01.842 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:59:01.842 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:01.842 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:02.170 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:59:02.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[100622]: from='client.62579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[100622]: from='client.62585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/522564839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:02.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[102844]: from='client.62579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[102844]: from='client.62585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/522564839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:02 smithi196 ceph-mon[76349]: from='client.62579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:02 smithi196 ceph-mon[76349]: from='client.62585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/522564839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[100622]: from='client.62597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[100622]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[102844]: from='client.62597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:03 smithi184 ceph-mon[102844]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:03 smithi196 ceph-mon[76349]: from='client.62597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:03 smithi196 ceph-mon[76349]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:04.102 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:04.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:04.102 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:04.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:04.102 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:04.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:05 smithi196 ceph-mon[76349]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:05 smithi184 ceph-mon[100622]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:05 smithi184 ceph-mon[102844]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:07 smithi196 ceph-mon[76349]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:07 smithi184 ceph-mon[100622]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:07 smithi184 ceph-mon[102844]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:09.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:09 smithi196 ceph-mon[76349]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:09 smithi184 ceph-mon[100622]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:09 smithi184 ceph-mon[102844]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:10.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:10.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:10.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:11.736 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:11 smithi196 ceph-mon[76349]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:11.736 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[102844]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[100622]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T21:59:13.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:13.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:13 smithi196 ceph-mon[76349]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[100622]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[102844]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:14.403 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:14.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:14.403 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:14.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:14.404 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:14.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:15 smithi184 ceph-mon[100622]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:15 smithi184 ceph-mon[102844]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:15 smithi196 ceph-mon[76349]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:17 smithi184 ceph-mon[100622]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:17 smithi184 ceph-mon[102844]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:17 smithi196 ceph-mon[76349]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:19 smithi184 ceph-mon[100622]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:19 smithi184 ceph-mon[102844]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:19 smithi196 ceph-mon[76349]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:20.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:20.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:20.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:21 smithi184 ceph-mon[100622]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:21 smithi184 ceph-mon[102844]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:21 smithi196 ceph-mon[76349]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:59:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:23.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:23 smithi184 ceph-mon[102844]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:23.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:23 smithi184 ceph-mon[100622]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:23 smithi196 ceph-mon[76349]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:24.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:24.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:24.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:24.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:24.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:24.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:25.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:25 smithi184 ceph-mon[100622]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:25.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:25 smithi184 ceph-mon[102844]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:25 smithi196 ceph-mon[76349]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:27 smithi184 ceph-mon[100622]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:27 smithi184 ceph-mon[102844]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:27 smithi196 ceph-mon[76349]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:29 smithi184 ceph-mon[100622]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:29 smithi184 ceph-mon[102844]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:29.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:29 smithi196 ceph-mon[76349]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:30.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:30.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:30.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:30.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:30.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:30.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:31 smithi196 ceph-mon[76349]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[102844]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[100622]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:32.141 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T21:59:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 2h - - 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 0s ago 2h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T21:59:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 2h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (119m) 1s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (118m) 0s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (117m) 0s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (116m) 0s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (115m) 0s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T21:59:32.675 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T21:59:32.676 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (115m) 1s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T21:59:32.676 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (115m) 0s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T21:59:32.676 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (115m) 1s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T21:59:32.676 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (115m) 0s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T21:59:32.984 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:59:32.984 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T21:59:32.985 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T21:59:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='client.62609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2168868085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='client.62609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2168868085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:59:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:33.562 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:33.562 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='client.62609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2168868085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:34.442 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:34.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:34.442 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:34.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:34.442 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:34.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[100622]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[100622]: from='client.52665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3912045832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[102844]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[102844]: from='client.52665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3912045832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:34 smithi196 ceph-mon[76349]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:34 smithi196 ceph-mon[76349]: from='client.52665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3912045832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:35 smithi184 ceph-mon[100622]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:35 smithi184 ceph-mon[102844]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:35 smithi196 ceph-mon[76349]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:37 smithi184 ceph-mon[100622]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:37 smithi184 ceph-mon[102844]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:37 smithi196 ceph-mon[76349]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:39 smithi184 ceph-mon[100622]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:39 smithi184 ceph-mon[102844]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:39 smithi196 ceph-mon[76349]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:40.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:40.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:40.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:41 smithi184 ceph-mon[100622]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:41 smithi184 ceph-mon[102844]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:41 smithi196 ceph-mon[76349]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T21:59:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:43 smithi184 ceph-mon[100622]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:43.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:43.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:43 smithi184 ceph-mon[102844]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:43 smithi196 ceph-mon[76349]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:44.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:44.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:44.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:44.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:44.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:45 smithi184 ceph-mon[100622]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:45 smithi184 ceph-mon[102844]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:45 smithi196 ceph-mon[76349]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:47 smithi184 ceph-mon[100622]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:47 smithi184 ceph-mon[102844]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:47 smithi196 ceph-mon[76349]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:49 smithi184 ceph-mon[100622]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:49 smithi184 ceph-mon[102844]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:49 smithi196 ceph-mon[76349]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:50.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:50.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:50.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[100622]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[102844]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:51 smithi196 ceph-mon[76349]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:51.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:51.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 21:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:21:59:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T21:59:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[100622]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[102844]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:53.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:53 smithi196 ceph-mon[76349]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:53.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:53.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T21:59:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:54.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:54.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T21:59:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 21:59:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T21:59:54.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T21:59:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:55 smithi184 ceph-mon[100622]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:55 smithi184 ceph-mon[102844]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:55 smithi196 ceph-mon[76349]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:57 smithi184 ceph-mon[100622]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:57 smithi184 ceph-mon[102844]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:57 smithi196 ceph-mon[76349]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:59 smithi184 ceph-mon[100622]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 21:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:59 smithi184 ceph-mon[102844]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 21:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:59 smithi196 ceph-mon[76349]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 21:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T21:59:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:00.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:00.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:00.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:01 smithi184 ceph-mon[102844]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:01 smithi184 ceph-mon[100622]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:01 smithi196 ceph-mon[76349]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:00:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:02 smithi184 ceph-mon[100622]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:02 smithi184 ceph-mon[102844]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:02 smithi196 ceph-mon[76349]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:03.861 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:00:04.347 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:04.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:04.347 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:04.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:04.347 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:04.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:00:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 2h - - 2024-09-20T22:00:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 2h - - 2024-09-20T22:00:04.459 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 13s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 13s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 13s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 13s ago 2h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 11s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 13s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 13s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 13s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 13s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:00:04.460 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (118m) 11s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (118m) 11s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (117m) 11s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (116m) 11s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (116m) 13s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (115m) 11s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (115m) 13s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:00:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (115m) 11s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:00:04.784 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:00:04.785 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:00:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:00:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:00:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:00:05.060 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:00:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[102844]: from='client.62645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[102844]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[102844]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[102844]: from='client.62657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/43295214' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[100622]: from='client.62645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[100622]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[100622]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[100622]: from='client.62657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/43295214' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:05.377 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:05.378 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:05.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:05.378 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:05.378 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:04 smithi196 ceph-mon[76349]: from='client.62645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:04 smithi196 ceph-mon[76349]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:04 smithi196 ceph-mon[76349]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:04 smithi196 ceph-mon[76349]: from='client.62657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/43295214' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:06.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:06 smithi184 ceph-mon[100622]: from='client.52701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:06.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2934358179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:06.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:06 smithi184 ceph-mon[102844]: from='client.52701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:06.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2934358179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:06 smithi196 ceph-mon[76349]: from='client.52701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2934358179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:07 smithi184 ceph-mon[100622]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:07 smithi184 ceph-mon[102844]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:07 smithi196 ceph-mon[76349]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:09 smithi184 ceph-mon[100622]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:09 smithi184 ceph-mon[102844]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:09 smithi196 ceph-mon[76349]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:11.014 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:10.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:11.014 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:10.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:11.014 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:10.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:11 smithi184 ceph-mon[100622]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:11 smithi184 ceph-mon[102844]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:11.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:11 smithi196 ceph-mon[76349]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:00:12.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:12.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:13.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:13 smithi196 ceph-mon[76349]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:13.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[100622]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[102844]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:14.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:14.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:14.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:14.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:14.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:15 smithi184 ceph-mon[100622]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:15 smithi184 ceph-mon[102844]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:15 smithi196 ceph-mon[76349]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:17 smithi184 ceph-mon[100622]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:17 smithi184 ceph-mon[102844]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:17 smithi196 ceph-mon[76349]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:19 smithi184 ceph-mon[100622]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:19 smithi184 ceph-mon[102844]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:19 smithi196 ceph-mon[76349]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:20.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:20.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:20.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:21 smithi184 ceph-mon[100622]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:21 smithi184 ceph-mon[102844]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:21 smithi196 ceph-mon[76349]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:00:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:23 smithi184 ceph-mon[100622]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:23 smithi184 ceph-mon[102844]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:23 smithi196 ceph-mon[76349]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:24.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:24.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:24.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:24.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:25 smithi184 ceph-mon[100622]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:25 smithi184 ceph-mon[102844]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:25 smithi196 ceph-mon[76349]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:27 smithi184 ceph-mon[100622]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:27 smithi184 ceph-mon[102844]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:27 smithi196 ceph-mon[76349]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:29 smithi184 ceph-mon[100622]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:29 smithi184 ceph-mon[102844]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:29 smithi196 ceph-mon[76349]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:30.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:30.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:30.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:31 smithi184 ceph-mon[100622]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:31 smithi184 ceph-mon[102844]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:31 smithi196 ceph-mon[76349]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:00:32.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[102844]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[100622]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:33 smithi196 ceph-mon[76349]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:34.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:34.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:34.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:35 smithi196 ceph-mon[76349]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:35.675 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:35 smithi184 ceph-mon[100622]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:35.676 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:35 smithi184 ceph-mon[102844]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:35.676 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 2h - - 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 2h - - 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 3s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:00:36.204 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 2h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 3s ago 2h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 4s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 4s ago 2h 74.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 4s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (119m) 3s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (118m) 3s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (117m) 3s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:00:36.205 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (116m) 3s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:00:36.206 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:00:36.206 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (116m) 4s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:00:36.206 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (116m) 3s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:00:36.206 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (116m) 4s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:00:36.206 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (116m) 3s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:00:36.516 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:00:36.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:00:36.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:00:36.518 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:00:36.784 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:00:36.784 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:00:36.784 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:00:36.784 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:00:36.784 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:00:36.785 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:00:37.097 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:37.097 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:37.098 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: from='client.62681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: from='client.62687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:37.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: from='client.62693 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3828364643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:37.698 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1184302171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: from='client.62681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: from='client.62687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: from='client.62693 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3828364643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1184302171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: from='client.62681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: from='client.62687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: from='client.62693 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3828364643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1184302171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:38 smithi196 ceph-mon[76349]: from='client.62705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:38 smithi184 ceph-mon[100622]: from='client.62705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:38 smithi184 ceph-mon[102844]: from='client.62705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:39 smithi196 ceph-mon[76349]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:39 smithi184 ceph-mon[100622]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:39 smithi184 ceph-mon[102844]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:40.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:40.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:40.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:41 smithi196 ceph-mon[76349]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:41.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:41 smithi184 ceph-mon[100622]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:41.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:41 smithi184 ceph-mon[102844]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:00:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:43 smithi184 ceph-mon[100622]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:43 smithi184 ceph-mon[102844]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:43 smithi196 ceph-mon[76349]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:44.385 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:44.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:44.386 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:44.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:44.386 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:44.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:45 smithi196 ceph-mon[76349]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:45.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:45 smithi184 ceph-mon[100622]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:45.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:45 smithi184 ceph-mon[102844]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:47.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:47 smithi184 ceph-mon[100622]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:47.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:47 smithi184 ceph-mon[102844]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:47 smithi196 ceph-mon[76349]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:49 smithi184 ceph-mon[100622]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:49 smithi184 ceph-mon[102844]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:49 smithi196 ceph-mon[76349]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:50.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:50.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:50.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:51.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:51 smithi184 ceph-mon[100622]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:51.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:51 smithi184 ceph-mon[102844]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:51 smithi196 ceph-mon[76349]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:00:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:00:52.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:53.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:53 smithi196 ceph-mon[76349]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:53.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.754 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[100622]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[102844]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:00:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:54.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:54.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:00:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:00:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:00:54.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:00:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:55 smithi184 ceph-mon[100622]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:55 smithi184 ceph-mon[102844]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:55 smithi196 ceph-mon[76349]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:56 smithi196 ceph-mon[76349]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:56 smithi184 ceph-mon[100622]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:56 smithi184 ceph-mon[102844]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:59 smithi184 ceph-mon[100622]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:59 smithi184 ceph-mon[102844]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:59 smithi196 ceph-mon[76349]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:01.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:00.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:01.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:00.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:01.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:00.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:01 smithi184 ceph-mon[100622]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:01 smithi184 ceph-mon[102844]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:01 smithi196 ceph-mon[76349]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:01:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:03 smithi184 ceph-mon[100622]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:03 smithi184 ceph-mon[102844]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:03 smithi196 ceph-mon[76349]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:04.481 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:04.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:04.482 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:04.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:04.482 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:04.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:05 smithi184 ceph-mon[102844]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:05 smithi184 ceph-mon[100622]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:05 smithi196 ceph-mon[76349]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:07 smithi184 ceph-mon[100622]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:07 smithi184 ceph-mon[102844]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:07 smithi196 ceph-mon[76349]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:01:07.966 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 16s ago 2h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 16s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 16s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 16s ago 2h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 16s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 16s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 16s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:01:07.967 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 16s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 16s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 16s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (119m) 14s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (118m) 14s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (117m) 14s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (117m) 16s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (116m) 14s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (116m) 16s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:01:07.968 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (116m) 14s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:01:08.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:01:08.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:08.288 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:01:08.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:01:08.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:01:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:01:08.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:08 smithi184 ceph-mon[100622]: from='client.62717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:08.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:08 smithi184 ceph-mon[102844]: from='client.62717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:08 smithi196 ceph-mon[76349]: from='client.62717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:08.559 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:01:08.559 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:01:08.559 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:01:08.560 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:01:08.869 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:01:08.869 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:08.869 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:08.869 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:08.869 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[100622]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[100622]: from='client.62729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[100622]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2340134787' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3728853095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[102844]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[102844]: from='client.62729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[102844]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2340134787' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3728853095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:09 smithi196 ceph-mon[76349]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:09 smithi196 ceph-mon[76349]: from='client.62729 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:09 smithi196 ceph-mon[76349]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2340134787' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3728853095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:10 smithi184 ceph-mon[100622]: from='client.62741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:10 smithi184 ceph-mon[102844]: from='client.62741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:10 smithi196 ceph-mon[76349]: from='client.62741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:11.042 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:10.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:11.042 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:10.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:11.042 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:10.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:11 smithi184 ceph-mon[102844]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:11 smithi184 ceph-mon[100622]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:11 smithi196 ceph-mon[76349]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:12 smithi196 ceph-mon[76349]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[100622]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:12 smithi184 ceph-mon[102844]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:14.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:14.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:14.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:14.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:14.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:14.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:14.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:15 smithi196 ceph-mon[76349]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:15 smithi184 ceph-mon[100622]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:15 smithi184 ceph-mon[102844]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:17 smithi184 ceph-mon[100622]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:17 smithi184 ceph-mon[102844]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:17 smithi196 ceph-mon[76349]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:19 smithi184 ceph-mon[100622]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:19 smithi184 ceph-mon[102844]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:19 smithi196 ceph-mon[76349]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:20.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:20.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:20.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:21 smithi184 ceph-mon[100622]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:21 smithi184 ceph-mon[102844]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:21 smithi196 ceph-mon[76349]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:01:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:23 smithi184 ceph-mon[100622]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:23 smithi184 ceph-mon[102844]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:23 smithi196 ceph-mon[76349]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:24.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:24.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:24.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:25 smithi184 ceph-mon[100622]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:25 smithi184 ceph-mon[102844]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:25 smithi196 ceph-mon[76349]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:27 smithi196 ceph-mon[76349]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:27.496 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:27 smithi184 ceph-mon[100622]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:27.496 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:27 smithi184 ceph-mon[102844]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:29 smithi184 ceph-mon[100622]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:29 smithi184 ceph-mon[102844]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:29 smithi196 ceph-mon[76349]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:30.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:30.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:30.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:30.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:30.942 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:30.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:31 smithi196 ceph-mon[76349]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:31.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:31 smithi184 ceph-mon[100622]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:31.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:31 smithi184 ceph-mon[102844]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:01:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[102844]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:32 smithi184 ceph-mon[100622]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:32 smithi196 ceph-mon[76349]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:34.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:34.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:34.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:34.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:34.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:34.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:34.671 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.671 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:35 smithi196 ceph-mon[76349]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:35 smithi184 ceph-mon[102844]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:35 smithi184 ceph-mon[100622]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:37 smithi184 ceph-mon[100622]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:37 smithi184 ceph-mon[102844]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:37 smithi196 ceph-mon[76349]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:01:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:39 smithi184 ceph-mon[100622]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:39 smithi184 ceph-mon[102844]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 2h - - 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 6s ago 2h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 2h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 6s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 7s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 7s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 7s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (119m) 6s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (118m) 6s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (117m) 6s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (117m) 7s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (117m) 6s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (117m) 7s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:01:39.691 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (117m) 6s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:01:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:39 smithi196 ceph-mon[76349]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:40.000 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:01:40.000 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:01:40.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:01:40.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:01:40.001 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:01:40.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:01:40.268 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:01:40.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[100622]: from='client.52785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[100622]: from='client.52791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2238737577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[102844]: from='client.52785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[102844]: from='client.52791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2238737577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:40.580 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:01:40.580 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:40.580 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:40.580 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:40.580 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:40 smithi196 ceph-mon[76349]: from='client.52785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:40 smithi196 ceph-mon[76349]: from='client.52791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2238737577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:40.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:40.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:40.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:41.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[100622]: from='client.62765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[100622]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:41.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[100622]: from='client.52809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3041421652' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:41.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[102844]: from='client.62765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[102844]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:41.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[102844]: from='client.52809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3041421652' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:41 smithi196 ceph-mon[76349]: from='client.62765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:41 smithi196 ceph-mon[76349]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:41 smithi196 ceph-mon[76349]: from='client.52809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3041421652' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:01:43.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:43.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:43 smithi196 ceph-mon[76349]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:43 smithi184 ceph-mon[100622]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:43 smithi184 ceph-mon[102844]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:44.426 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:44.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:44.427 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:44.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:44.427 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:44.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:44.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:45 smithi184 ceph-mon[102844]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:45 smithi184 ceph-mon[100622]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:45 smithi196 ceph-mon[76349]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:47.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:47 smithi184 ceph-mon[100622]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:47.740 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:47 smithi184 ceph-mon[102844]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:47 smithi196 ceph-mon[76349]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:49.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:49 smithi184 ceph-mon[100622]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:49.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:49 smithi184 ceph-mon[102844]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:49 smithi196 ceph-mon[76349]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:01:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:50.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:50.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:50.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:51.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:51 smithi184 ceph-mon[100622]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:51.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:51 smithi184 ceph-mon[102844]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:51 smithi196 ceph-mon[76349]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:01:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:52 smithi196 ceph-mon[76349]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[100622]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:52 smithi184 ceph-mon[102844]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:54.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:54.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:54.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:54.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:01:54.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:01:54.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:01:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:54 smithi196 ceph-mon[76349]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[100622]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:01:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:54 smithi184 ceph-mon[102844]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:57 smithi184 ceph-mon[100622]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:57 smithi184 ceph-mon[102844]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:57.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:57 smithi196 ceph-mon[76349]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:59 smithi184 ceph-mon[102844]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:59 smithi184 ceph-mon[100622]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:59 smithi196 ceph-mon[76349]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:01.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:00.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:01.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:00.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:01.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:00.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:01 smithi184 ceph-mon[102844]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:01 smithi184 ceph-mon[100622]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:01 smithi196 ceph-mon[76349]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:02.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:02:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:03 smithi196 ceph-mon[76349]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:03 smithi184 ceph-mon[100622]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:03 smithi184 ceph-mon[102844]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:04.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:04.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:04.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:04.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:05 smithi196 ceph-mon[76349]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:05 smithi184 ceph-mon[100622]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:05 smithi184 ceph-mon[102844]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:07 smithi196 ceph-mon[76349]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:07 smithi184 ceph-mon[100622]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:07 smithi184 ceph-mon[102844]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:09.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:09 smithi184 ceph-mon[100622]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:09.341 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:09 smithi184 ceph-mon[102844]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:09 smithi196 ceph-mon[76349]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:10.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:10.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:10.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:10.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:10.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:10.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:10.870 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:02:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:11 smithi196 ceph-mon[76349]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:02:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T22:02:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 19s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 19s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 19s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 19s ago 2h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 19s ago 2h 183M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 19s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 19s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 19s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 19s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:02:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (119m) 17s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (118m) 17s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (118m) 19s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (117m) 17s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (118m) 19s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:02:11.407 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (117m) 17s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:02:11.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:11 smithi184 ceph-mon[100622]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:11.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:11 smithi184 ceph-mon[102844]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:11.714 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:02:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:02:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:02:11.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:02:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:02:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:02:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:02:12.006 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:02:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:02:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='client.62789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='client.62795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1891427260' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='client.62789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='client.62795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1891427260' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.317 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:02:12.317 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:12.317 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='client.62789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='client.62795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1891427260' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:13 smithi196 ceph-mon[76349]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:13 smithi196 ceph-mon[76349]: from='client.52845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3105962524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[102844]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[102844]: from='client.52845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3105962524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[100622]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[100622]: from='client.52845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3105962524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:14.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:14.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:14.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:14.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:14.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:14.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:14.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:14.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:14.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:14.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:14.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:14 smithi196 ceph-mon[76349]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[102844]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:14 smithi184 ceph-mon[100622]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:17 smithi184 ceph-mon[100622]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:17 smithi184 ceph-mon[102844]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:17 smithi196 ceph-mon[76349]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:19 smithi184 ceph-mon[100622]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:19 smithi184 ceph-mon[102844]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:19 smithi196 ceph-mon[76349]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:20.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:20.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:20.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:21 smithi196 ceph-mon[76349]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:21 smithi184 ceph-mon[100622]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:21 smithi184 ceph-mon[102844]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:02:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:23 smithi196 ceph-mon[76349]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:23 smithi184 ceph-mon[100622]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:23 smithi184 ceph-mon[102844]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:24.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:24.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:24.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:25 smithi196 ceph-mon[76349]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:25 smithi184 ceph-mon[100622]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:25 smithi184 ceph-mon[102844]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:27 smithi196 ceph-mon[76349]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:27 smithi184 ceph-mon[100622]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:27 smithi184 ceph-mon[102844]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:29.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:29 smithi184 ceph-mon[100622]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:29.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:29.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:29 smithi184 ceph-mon[102844]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:29.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:29 smithi196 ceph-mon[76349]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:30.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:30.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:30.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:31.379 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:31 smithi184 ceph-mon[100622]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:31.379 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:31 smithi184 ceph-mon[102844]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:31 smithi196 ceph-mon[76349]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:33 smithi196 ceph-mon[76349]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[100622]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[102844]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:34.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:34.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:34.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:34.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[100622]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:34 smithi184 ceph-mon[102844]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:35.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:35.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:34 smithi196 ceph-mon[76349]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:37 smithi196 ceph-mon[76349]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:37 smithi184 ceph-mon[100622]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:37 smithi184 ceph-mon[102844]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:39 smithi196 ceph-mon[76349]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:39 smithi184 ceph-mon[100622]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:39 smithi184 ceph-mon[102844]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:40.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:40.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:40.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:41 smithi196 ceph-mon[76349]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:41 smithi184 ceph-mon[100622]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:41 smithi184 ceph-mon[102844]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:02:42.612 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:02:43.143 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:02:43.143 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 2h - - 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 11s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 2h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 2h 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:02:43.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 2h 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (119m) 9s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (119m) 9s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (118m) 11s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (118m) 9s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (118m) 11s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:02:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (118m) 9s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:02:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:43 smithi196 ceph-mon[76349]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:02:43.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:43.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:43 smithi184 ceph-mon[100622]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:02:43.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:43.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:43 smithi184 ceph-mon[102844]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:43.458 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:02:43.459 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:02:43.729 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:02:43.729 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:02:43.729 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:02:43.729 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:02:43.729 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:02:43.730 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:02:44.046 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:02:44.046 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:44.047 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:44.047 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:44.047 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='client.62825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='client.52863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='client.52869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3845127851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3355712765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='client.62825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='client.52863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='client.52869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3845127851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3355712765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:44.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:44.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:44.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:44.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:44.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='client.62825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='client.52863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='client.52869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3845127851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3355712765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:45 smithi196 ceph-mon[76349]: from='client.62849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:45 smithi196 ceph-mon[76349]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:45 smithi184 ceph-mon[102844]: from='client.62849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:45 smithi184 ceph-mon[102844]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:45 smithi184 ceph-mon[100622]: from='client.62849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:45 smithi184 ceph-mon[100622]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:47 smithi196 ceph-mon[76349]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:47 smithi184 ceph-mon[100622]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:47 smithi184 ceph-mon[102844]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:49 smithi196 ceph-mon[76349]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:49.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:49 smithi184 ceph-mon[100622]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:49.504 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:49 smithi184 ceph-mon[102844]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:50.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:50.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:50.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:50.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:51 smithi184 ceph-mon[100622]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:51 smithi184 ceph-mon[102844]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:51 smithi196 ceph-mon[76349]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:52.134 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:02:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:02:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[100622]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[102844]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:53 smithi196 ceph-mon[76349]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:54.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:54.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:54.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:02:54.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:02:54.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[100622]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:54 smithi184 ceph-mon[102844]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:02:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:54 smithi196 ceph-mon[76349]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:57 smithi196 ceph-mon[76349]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:57 smithi184 ceph-mon[100622]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:57 smithi184 ceph-mon[102844]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:59 smithi196 ceph-mon[76349]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:59 smithi184 ceph-mon[102844]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:59 smithi184 ceph-mon[100622]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:00.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:00.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:00.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:01 smithi196 ceph-mon[76349]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:01 smithi184 ceph-mon[100622]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:01 smithi184 ceph-mon[102844]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:03:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:03 smithi196 ceph-mon[76349]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:03 smithi184 ceph-mon[100622]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:03 smithi184 ceph-mon[102844]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:04.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:04.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:04.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:05 smithi196 ceph-mon[76349]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:05 smithi184 ceph-mon[100622]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:05 smithi184 ceph-mon[102844]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:07 smithi196 ceph-mon[76349]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:07 smithi184 ceph-mon[100622]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:07 smithi184 ceph-mon[102844]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:09 smithi196 ceph-mon[76349]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:09 smithi184 ceph-mon[100622]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:09 smithi184 ceph-mon[102844]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:10.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:10.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:10.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:10.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:10.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:10.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:11.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:11 smithi184 ceph-mon[100622]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:11.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:11 smithi184 ceph-mon[102844]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:11 smithi196 ceph-mon[76349]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:12.092 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:03:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:13 smithi196 ceph-mon[76349]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[102844]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[100622]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.338 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:14.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:14.338 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:14.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:14.338 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:14.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:14.339 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:03:14.868 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 2h - - 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 2h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 0s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 2s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 2s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 2s ago 2h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 2s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 2s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 2s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 2s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:03:14.869 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 2s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 0s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 0s ago 2h 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (119m) 0s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (119m) 2s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (118m) 0s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (119m) 2s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:03:14.870 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (118m) 0s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:03:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:03:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:03:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:03:15.180 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:03:15.452 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:03:15.452 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:03:15.453 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[100622]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[100622]: from='client.62861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2565815662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[102844]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[102844]: from='client.62861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:15.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2565815662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:15 smithi196 ceph-mon[76349]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:15 smithi196 ceph-mon[76349]: from='client.62861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2565815662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:15.764 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:03:15.764 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:15.764 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:15.764 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:15.764 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:16.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:16 smithi196 ceph-mon[76349]: from='client.62867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:16 smithi196 ceph-mon[76349]: from='client.62873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:16 smithi196 ceph-mon[76349]: from='client.62885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3658268031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[100622]: from='client.62867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[100622]: from='client.62873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[100622]: from='client.62885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3658268031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[102844]: from='client.62867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[102844]: from='client.62873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[102844]: from='client.62885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3658268031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:17 smithi196 ceph-mon[76349]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:17 smithi184 ceph-mon[100622]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:17 smithi184 ceph-mon[102844]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:19 smithi184 ceph-mon[100622]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:19 smithi184 ceph-mon[102844]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:19 smithi196 ceph-mon[76349]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:20.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:20.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:20.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:21 smithi184 ceph-mon[100622]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:21 smithi184 ceph-mon[102844]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:21 smithi196 ceph-mon[76349]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:03:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:23 smithi184 ceph-mon[100622]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:23 smithi184 ceph-mon[102844]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:23 smithi196 ceph-mon[76349]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:24.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:24.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:24.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:24 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:24.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:25 smithi184 ceph-mon[100622]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:25 smithi184 ceph-mon[102844]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:25 smithi196 ceph-mon[76349]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:27 smithi184 ceph-mon[100622]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:27 smithi184 ceph-mon[102844]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:27 smithi196 ceph-mon[76349]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:29 smithi184 ceph-mon[102844]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:29 smithi184 ceph-mon[100622]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:29 smithi196 ceph-mon[76349]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:30.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:30.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:30.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:30.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:30.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:30.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:31 smithi184 ceph-mon[100622]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:31 smithi184 ceph-mon[102844]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:31 smithi196 ceph-mon[76349]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:32.104 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:03:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[100622]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[102844]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:33 smithi196 ceph-mon[76349]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:34.289 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:34.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:34.290 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:34.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:34.290 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:34 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:34.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[102844]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[100622]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:35 smithi196 ceph-mon[76349]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:37.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:37 smithi196 ceph-mon[76349]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:37 smithi184 ceph-mon[102844]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:37 smithi184 ceph-mon[100622]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:39.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:39 smithi196 ceph-mon[76349]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:39 smithi184 ceph-mon[102844]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:39 smithi184 ceph-mon[100622]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:40.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:40.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:40.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:41 smithi184 ceph-mon[100622]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:41 smithi184 ceph-mon[102844]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:41 smithi196 ceph-mon[76349]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:03:43.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:43.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:43 smithi196 ceph-mon[76349]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:43 smithi184 ceph-mon[100622]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:43 smithi184 ceph-mon[102844]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:44.463 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:44.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:44.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:44.522 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:44.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:44.522 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:44.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:44.522 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:44 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:44.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:45 smithi184 ceph-mon[100622]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:45 smithi184 ceph-mon[102844]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:45.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:45 smithi196 ceph-mon[76349]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:46.066 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 12s ago 2h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 2h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:03:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 12s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 12s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 12s ago 2h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 12s ago 2h 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 12s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 12s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 12s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (119m) 14s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (119m) 12s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:03:46.592 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (119m) 14s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:03:46.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (119m) 12s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:03:46.901 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:03:46.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:03:46.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:03:46.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:03:46.902 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:03:47.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:03:47.170 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:03:47.171 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:03:47.485 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:03:47.485 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:47.485 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:47.485 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:47.485 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:47.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[100622]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:47.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[100622]: from='client.62897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[100622]: from='client.52935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2649605084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:47.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[102844]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:47.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[102844]: from='client.62897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[102844]: from='client.52935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2649605084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:47 smithi196 ceph-mon[76349]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:47 smithi196 ceph-mon[76349]: from='client.62897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:47 smithi196 ceph-mon[76349]: from='client.52935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2649605084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[100622]: from='client.52941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[100622]: from='client.52953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1370296506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[102844]: from='client.52941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[102844]: from='client.52953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1370296506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:48 smithi196 ceph-mon[76349]: from='client.52941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:48 smithi196 ceph-mon[76349]: from='client.52953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1370296506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:49 smithi184 ceph-mon[100622]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:49 smithi184 ceph-mon[102844]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:49 smithi196 ceph-mon[76349]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:50.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:50.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:50.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:51.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:51 smithi184 ceph-mon[100622]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:51.504 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:51 smithi184 ceph-mon[102844]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:51 smithi196 ceph-mon[76349]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:03:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:03:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[100622]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[102844]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:53 smithi196 ceph-mon[76349]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:54.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:54.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:03:54.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:03:54 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:03:54.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:03:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[100622]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[102844]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:55 smithi196 ceph-mon[76349]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:03:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:57 smithi184 ceph-mon[100622]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:57 smithi184 ceph-mon[102844]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:57 smithi196 ceph-mon[76349]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:59.413 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:59 smithi196 ceph-mon[76349]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:59.413 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:59 smithi184 ceph-mon[100622]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:59 smithi184 ceph-mon[102844]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:00.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:00.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:00.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:01 smithi196 ceph-mon[76349]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:01 smithi184 ceph-mon[100622]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:01 smithi184 ceph-mon[102844]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:04:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:03 smithi196 ceph-mon[76349]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:03 smithi184 ceph-mon[100622]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:03 smithi184 ceph-mon[102844]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:04.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:04.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:04.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:04 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:04.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:05 smithi196 ceph-mon[76349]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:05 smithi184 ceph-mon[100622]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:05 smithi184 ceph-mon[102844]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:07 smithi196 ceph-mon[76349]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:04:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:07 smithi184 ceph-mon[100622]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:04:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:07 smithi184 ceph-mon[102844]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:04:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:09 smithi196 ceph-mon[76349]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:04:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:09 smithi184 ceph-mon[100622]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:04:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:09 smithi184 ceph-mon[102844]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:04:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:10.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:10.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:10.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:11 smithi196 ceph-mon[76349]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T22:04:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:11 smithi184 ceph-mon[100622]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T22:04:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:11 smithi184 ceph-mon[102844]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T22:04:12.167 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:04:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[100622]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T22:04:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[102844]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T22:04:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:13 smithi196 ceph-mon[76349]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T22:04:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:14.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:14.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:14.401 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:14 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:14.089Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[100622]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[102844]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:15 smithi196 ceph-mon[76349]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:17.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:17 smithi184 ceph-mon[100622]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T22:04:17.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:17 smithi184 ceph-mon[102844]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T22:04:17.816 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:04:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:17 smithi196 ceph-mon[76349]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 2h - - 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 3s ago 2h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:04:18.306 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 2h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 2h 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 2h 183M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 3s ago 2h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 2h 65.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 3s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 3s ago 2h 69.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 2h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:04:18.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:04:18.308 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (119m) 3s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:04:18.615 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:04:18.615 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:04:18.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:04:18.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:04:18.617 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:04:18.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:04:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:04:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:04:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:04:18.883 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:04:18.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:04:19.192 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:04:19.192 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:19.192 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:19.192 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:19.192 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: from='client.52965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: from='client.62936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: from='client.62942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2101442154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2281693621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: from='client.52965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: from='client.62936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: from='client.62942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2101442154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2281693621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: from='client.52965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: from='client.62936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: from='client.62942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2101442154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:19.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2281693621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:20.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:20 smithi196 ceph-mon[76349]: from='client.62954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:20 smithi184 ceph-mon[100622]: from='client.62954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:20 smithi184 ceph-mon[102844]: from='client.62954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:20.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:20.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:20.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:20.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:21 smithi184 ceph-mon[100622]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:04:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:21 smithi184 ceph-mon[102844]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:04:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:21 smithi196 ceph-mon[76349]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:04:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:23 smithi184 ceph-mon[100622]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T22:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:23 smithi184 ceph-mon[102844]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T22:04:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:23 smithi196 ceph-mon[76349]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T22:04:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:25 smithi184 ceph-mon[100622]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:25 smithi184 ceph-mon[102844]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:25 smithi196 ceph-mon[76349]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:27 smithi184 ceph-mon[100622]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T22:04:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:27 smithi184 ceph-mon[102844]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T22:04:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:27 smithi196 ceph-mon[76349]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T22:04:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:29 smithi184 ceph-mon[100622]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T22:04:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:29 smithi184 ceph-mon[102844]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T22:04:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:29 smithi196 ceph-mon[76349]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T22:04:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:30.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:30.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:30.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:31 smithi184 ceph-mon[100622]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T22:04:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:31 smithi184 ceph-mon[102844]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T22:04:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:31 smithi196 ceph-mon[76349]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T22:04:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:04:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[100622]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[102844]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:33 smithi196 ceph-mon[76349]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T22:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[100622]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:04:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[102844]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:04:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:35 smithi196 ceph-mon[76349]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:37 smithi184 ceph-mon[100622]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:04:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:37 smithi184 ceph-mon[102844]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:04:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:37 smithi196 ceph-mon[76349]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:04:39.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:39 smithi196 ceph-mon[76349]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:04:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:39 smithi184 ceph-mon[100622]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:04:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:39 smithi184 ceph-mon[102844]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:04:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:40.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:40.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:40.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:41 smithi196 ceph-mon[76349]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:04:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:41 smithi184 ceph-mon[100622]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:04:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:41 smithi184 ceph-mon[102844]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:04:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:04:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:43 smithi184 ceph-mon[100622]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T22:04:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:43 smithi184 ceph-mon[102844]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T22:04:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:43 smithi196 ceph-mon[76349]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T22:04:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:45 smithi184 ceph-mon[100622]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:45 smithi184 ceph-mon[102844]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:45 smithi196 ceph-mon[76349]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:47 smithi184 ceph-mon[100622]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:47 smithi184 ceph-mon[102844]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:47 smithi196 ceph-mon[76349]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:49.488 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:04:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:49 smithi184 ceph-mon[100622]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:49 smithi184 ceph-mon[102844]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:49 smithi196 ceph-mon[76349]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:50.019 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:04:50.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T22:04:50.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T22:04:50.019 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 15s ago 2h 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 2h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 15s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 15s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:04:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 2h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 2h 66.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 2h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 15s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 15s ago 2h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 15s ago 2h 73.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 15s ago 2h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 15s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:04:50.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 15s ago 2h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:04:50.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:04:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:04:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:04:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:04:50.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:04:50.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-mon[100622]: from='client.62966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:50.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3740860033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:50.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-mon[102844]: from='client.62966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:50.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3740860033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:04:50.608 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:04:50.609 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:04:50.609 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:04:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:50 smithi196 ceph-mon[76349]: from='client.62966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3740860033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:50.916 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:04:50.916 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:50.916 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:50.916 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:50.916 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:50.917 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:50.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:50.917 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:50.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:04:50.917 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:04:50.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:04:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[102844]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[102844]: from='client.53013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[102844]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/229943940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[100622]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[100622]: from='client.53013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[100622]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/229943940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:51.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:51 smithi196 ceph-mon[76349]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:51 smithi196 ceph-mon[76349]: from='client.53013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:51.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:51 smithi196 ceph-mon[76349]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/229943940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:04:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:04:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:52 smithi196 ceph-mon[76349]: from='client.53025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:52 smithi184 ceph-mon[100622]: from='client.53025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:52 smithi184 ceph-mon[102844]: from='client.53025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:53.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:53.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[100622]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[102844]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:53 smithi196 ceph-mon[76349]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[100622]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[102844]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:55 smithi196 ceph-mon[76349]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:04:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:57 smithi184 ceph-mon[100622]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:57 smithi184 ceph-mon[102844]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:57 smithi196 ceph-mon[76349]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:59 smithi184 ceph-mon[100622]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:59 smithi184 ceph-mon[102844]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:59 smithi196 ceph-mon[76349]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:00.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:00.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:00.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:01 smithi196 ceph-mon[76349]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:01 smithi184 ceph-mon[100622]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:01 smithi184 ceph-mon[102844]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:05:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:03 smithi184 ceph-mon[100622]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:03 smithi184 ceph-mon[102844]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:04.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:03 smithi196 ceph-mon[76349]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:05.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:05 smithi184 ceph-mon[100622]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:05.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:05 smithi184 ceph-mon[102844]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:05 smithi196 ceph-mon[76349]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:07 smithi184 ceph-mon[102844]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:07 smithi184 ceph-mon[100622]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:07 smithi196 ceph-mon[76349]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:09 smithi184 ceph-mon[102844]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:09 smithi184 ceph-mon[100622]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:09 smithi196 ceph-mon[76349]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:10.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:10.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:10.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:11 smithi184 ceph-mon[100622]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:11 smithi184 ceph-mon[102844]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:11 smithi196 ceph-mon[76349]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:05:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[100622]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[102844]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:13 smithi196 ceph-mon[76349]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:14.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[100622]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[102844]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:15.672 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:15 smithi196 ceph-mon[76349]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:17 smithi184 ceph-mon[100622]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:17 smithi184 ceph-mon[102844]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:17 smithi196 ceph-mon[76349]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:19 smithi184 ceph-mon[100622]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:19 smithi184 ceph-mon[102844]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:19 smithi196 ceph-mon[76349]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:20.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:20.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:20.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:21.216 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 2h - - 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 6s ago 2h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 2h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:05:21.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 6s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 2h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 6s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 6s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 6s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:05:21.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:05:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:21 smithi184 ceph-mon[100622]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:21 smithi184 ceph-mon[102844]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:21.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:21 smithi196 ceph-mon[76349]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:22.060 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:05:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:05:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:05:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:05:22.061 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:05:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:05:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:05:22.329 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:05:22.635 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:05:22.635 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:22.635 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:22.635 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:22.635 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[100622]: from='client.63002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[100622]: from='client.63008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4224609932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[102844]: from='client.63002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[102844]: from='client.63008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4224609932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:22 smithi196 ceph-mon[76349]: from='client.63002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:22 smithi196 ceph-mon[76349]: from='client.63008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4224609932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[100622]: from='client.63014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[100622]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[100622]: from='client.63026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/432399229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[102844]: from='client.63014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[102844]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[102844]: from='client.63026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/432399229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:23 smithi196 ceph-mon[76349]: from='client.63014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:23 smithi196 ceph-mon[76349]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:23 smithi196 ceph-mon[76349]: from='client.63026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/432399229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:24 smithi196 ceph-mon[76349]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:24 smithi184 ceph-mon[100622]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:24 smithi184 ceph-mon[102844]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:27 smithi196 ceph-mon[76349]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:27 smithi184 ceph-mon[100622]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:27 smithi184 ceph-mon[102844]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:29 smithi196 ceph-mon[76349]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:29 smithi184 ceph-mon[100622]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:29 smithi184 ceph-mon[102844]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:30.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:30.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:30.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:31 smithi196 ceph-mon[76349]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:31 smithi184 ceph-mon[100622]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:31 smithi184 ceph-mon[102844]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:05:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:33 smithi196 ceph-mon[76349]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:33 smithi184 ceph-mon[100622]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:33.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:33.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:33 smithi184 ceph-mon[102844]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[100622]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:35.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[102844]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:35.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:35 smithi196 ceph-mon[76349]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:37 smithi184 ceph-mon[102844]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:37 smithi184 ceph-mon[100622]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:37 smithi196 ceph-mon[76349]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:39 smithi196 ceph-mon[76349]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:39 smithi184 ceph-mon[100622]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:39 smithi184 ceph-mon[102844]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:40.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:40.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:40.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:41.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:41 smithi196 ceph-mon[76349]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:41 smithi184 ceph-mon[100622]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:41 smithi184 ceph-mon[102844]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:05:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:43 smithi196 ceph-mon[76349]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:43 smithi184 ceph-mon[100622]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:43 smithi184 ceph-mon[102844]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:45 smithi196 ceph-mon[76349]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:45 smithi184 ceph-mon[100622]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:45 smithi184 ceph-mon[102844]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:47 smithi196 ceph-mon[76349]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:47 smithi184 ceph-mon[100622]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:47 smithi184 ceph-mon[102844]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:49 smithi196 ceph-mon[76349]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:49 smithi184 ceph-mon[100622]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:49 smithi184 ceph-mon[102844]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:50.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:50.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:05:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:05:50.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:51 smithi196 ceph-mon[76349]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:51 smithi184 ceph-mon[100622]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:51 smithi184 ceph-mon[102844]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:05:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:05:52.931 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:05:53.468 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 2h - - 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 0s ago 2h 29.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 0s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 2h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 0s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:05:53.469 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:05:53.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:05:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:53 smithi196 ceph-mon[76349]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:53.778 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:05:53.778 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:05:53.778 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:05:53.779 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:05:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:53.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:53 smithi184 ceph-mon[100622]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:53.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:53.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:53 smithi184 ceph-mon[102844]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:54.048 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:05:54.049 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:05:54.050 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:05:54.359 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:05:54.359 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:54.359 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:54.360 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:54.360 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:54.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='client.53073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='client.63044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='client.53073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='client.63050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1867787316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2613630095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='client.63044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='client.63050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1867787316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2613630095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='client.53073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='client.63044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='client.63050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1867787316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2613630095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[100622]: from='client.63062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[100622]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[102844]: from='client.63062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[102844]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:55.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:55 smithi196 ceph-mon[76349]: from='client.63062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:55 smithi196 ceph-mon[76349]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:05:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:57 smithi184 ceph-mon[100622]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:57 smithi184 ceph-mon[102844]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:57 smithi196 ceph-mon[76349]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:59 smithi184 ceph-mon[100622]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:59 smithi184 ceph-mon[102844]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:59 smithi196 ceph-mon[76349]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:00.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:00.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:00.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:01.781 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:01 smithi196 ceph-mon[76349]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:01 smithi184 ceph-mon[100622]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:01 smithi184 ceph-mon[102844]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:06:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:03 smithi184 ceph-mon[102844]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:03 smithi184 ceph-mon[100622]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:03 smithi196 ceph-mon[76349]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:05 smithi184 ceph-mon[102844]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:05 smithi184 ceph-mon[100622]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:05 smithi196 ceph-mon[76349]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:07 smithi184 ceph-mon[102844]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:07 smithi184 ceph-mon[100622]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:07 smithi196 ceph-mon[76349]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:09 smithi184 ceph-mon[100622]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:09 smithi184 ceph-mon[102844]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:09 smithi196 ceph-mon[76349]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:10.669Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:10.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:10.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:11 smithi184 ceph-mon[100622]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:11 smithi184 ceph-mon[102844]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:11 smithi196 ceph-mon[76349]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:06:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:13 smithi184 ceph-mon[100622]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:06:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:13 smithi184 ceph-mon[102844]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:06:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:13 smithi196 ceph-mon[76349]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:06:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:14 smithi196 ceph-mon[76349]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[100622]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:14 smithi184 ceph-mon[102844]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:16.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.208 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:17.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:17 smithi184 ceph-mon[100622]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:17.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:17 smithi184 ceph-mon[102844]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:17 smithi196 ceph-mon[76349]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:19 smithi184 ceph-mon[100622]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:19 smithi184 ceph-mon[102844]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:19 smithi196 ceph-mon[76349]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:20.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:20.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:20.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:21 smithi184 ceph-mon[100622]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:21 smithi184 ceph-mon[102844]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:21 smithi196 ceph-mon[76349]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:06:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:23 smithi196 ceph-mon[76349]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:23 smithi184 ceph-mon[100622]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:23 smithi184 ceph-mon[102844]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:24.658 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:06:25.182 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:06:25.182 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T22:06:25.182 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T22:06:25.182 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 11s ago 2h 29.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 2h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 2h 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:06:25.183 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 10s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 11s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 11s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:06:25.184 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:06:25.492 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:06:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:06:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:06:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:25.492 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:06:25.493 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:06:25.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:25 smithi184 ceph-mon[100622]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:25.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:25 smithi184 ceph-mon[102844]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:25 smithi196 ceph-mon[76349]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:06:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:06:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:06:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:06:25.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:06:26.076 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:06:26.076 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:26.076 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[100622]: from='client.63074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[100622]: from='client.63080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[100622]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3051568360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/965612636' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[102844]: from='client.63074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[102844]: from='client.63080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[102844]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3051568360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/965612636' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:26 smithi196 ceph-mon[76349]: from='client.63074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:26 smithi196 ceph-mon[76349]: from='client.63080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:26 smithi196 ceph-mon[76349]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3051568360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/965612636' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:27 smithi184 ceph-mon[100622]: from='client.53133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:27 smithi184 ceph-mon[100622]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:27 smithi184 ceph-mon[102844]: from='client.53133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:27 smithi184 ceph-mon[102844]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:27 smithi196 ceph-mon[76349]: from='client.53133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:27 smithi196 ceph-mon[76349]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:29 smithi184 ceph-mon[100622]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:29 smithi184 ceph-mon[102844]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:29 smithi196 ceph-mon[76349]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:30.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:30.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:30.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:31 smithi184 ceph-mon[100622]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:31 smithi184 ceph-mon[102844]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:31 smithi196 ceph-mon[76349]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:06:33.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:33.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:33 smithi184 ceph-mon[100622]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:33.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:33.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:33 smithi184 ceph-mon[102844]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:33 smithi196 ceph-mon[76349]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[100622]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:34 smithi184 ceph-mon[102844]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:34 smithi196 ceph-mon[76349]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:36.567 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.567 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.567 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:37 smithi184 ceph-mon[102844]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:37 smithi184 ceph-mon[100622]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:37 smithi196 ceph-mon[76349]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:39 smithi196 ceph-mon[76349]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:39 smithi184 ceph-mon[100622]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:39 smithi184 ceph-mon[102844]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:40.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:40.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:40.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:41 smithi196 ceph-mon[76349]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:41 smithi184 ceph-mon[100622]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:41 smithi184 ceph-mon[102844]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:06:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:43 smithi196 ceph-mon[76349]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:43 smithi184 ceph-mon[100622]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:43 smithi184 ceph-mon[102844]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:45 smithi196 ceph-mon[76349]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:45 smithi184 ceph-mon[100622]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:45 smithi184 ceph-mon[102844]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:47 smithi196 ceph-mon[76349]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:47 smithi184 ceph-mon[100622]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:47 smithi184 ceph-mon[102844]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:49 smithi196 ceph-mon[76349]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:49 smithi184 ceph-mon[100622]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:49 smithi184 ceph-mon[102844]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:50.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:50.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:06:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:06:50.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:06:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:51 smithi196 ceph-mon[76349]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:51 smithi184 ceph-mon[100622]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:51 smithi184 ceph-mon[102844]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:06:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:06:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:53 smithi196 ceph-mon[76349]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:53 smithi184 ceph-mon[100622]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:53 smithi184 ceph-mon[102844]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:54 smithi196 ceph-mon[76349]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[100622]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:54 smithi184 ceph-mon[102844]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:56.372 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:06:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:06:56.899 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 2h - - 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 3s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 3s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 3s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 2h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 2h 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 3s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:06:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:06:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:06:57.496 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[102844]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:57.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[102844]: from='client.53145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:57.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3507102249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:57.497 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[100622]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:57.497 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[100622]: from='client.53145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:57.497 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3507102249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:57.497 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:06:57.497 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:06:57.498 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:06:57.814 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:06:57.814 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:57.814 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:57.814 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:57.814 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:57 smithi196 ceph-mon[76349]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:57 smithi196 ceph-mon[76349]: from='client.53145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3507102249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[100622]: from='client.63116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[100622]: from='client.63122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[100622]: from='client.53169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2625248460' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[102844]: from='client.63116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[102844]: from='client.63122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[102844]: from='client.53169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2625248460' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:58 smithi196 ceph-mon[76349]: from='client.63116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:58 smithi196 ceph-mon[76349]: from='client.63122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:58 smithi196 ceph-mon[76349]: from='client.53169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2625248460' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:59.576 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:59 smithi184 ceph-mon[100622]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:59.576 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:59.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:59 smithi184 ceph-mon[102844]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:59.576 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:59 smithi196 ceph-mon[76349]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:00.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:00.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:00.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:01 smithi184 ceph-mon[100622]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:01 smithi184 ceph-mon[102844]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:01 smithi196 ceph-mon[76349]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:07:03.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:03.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:03 smithi196 ceph-mon[76349]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:03 smithi184 ceph-mon[100622]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:03 smithi184 ceph-mon[102844]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:05 smithi184 ceph-mon[100622]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:05 smithi184 ceph-mon[102844]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:05 smithi196 ceph-mon[76349]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:07 smithi184 ceph-mon[100622]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:07 smithi184 ceph-mon[102844]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:07 smithi196 ceph-mon[76349]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:09 smithi184 ceph-mon[100622]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:09 smithi184 ceph-mon[102844]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:09 smithi196 ceph-mon[76349]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:10.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:10.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:10.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:11 smithi184 ceph-mon[100622]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:11 smithi184 ceph-mon[102844]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:11 smithi196 ceph-mon[76349]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:07:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:13 smithi184 ceph-mon[102844]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:13 smithi184 ceph-mon[100622]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:13 smithi196 ceph-mon[76349]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:14.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:14.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:15 smithi184 ceph-mon[100622]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:15 smithi184 ceph-mon[102844]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:15 smithi196 ceph-mon[76349]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:17.833 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:17 smithi184 ceph-mon[100622]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:17.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:17 smithi184 ceph-mon[102844]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:17 smithi196 ceph-mon[76349]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:19.585 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:19 smithi184 ceph-mon[100622]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:19.585 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:19 smithi184 ceph-mon[102844]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:19 smithi196 ceph-mon[76349]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:20 smithi196 ceph-mon[76349]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:20 smithi184 ceph-mon[100622]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:20 smithi184 ceph-mon[102844]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:20.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:20.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:20.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:07:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:23 smithi184 ceph-mon[100622]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:23 smithi184 ceph-mon[102844]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:23 smithi196 ceph-mon[76349]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:25.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:25 smithi196 ceph-mon[76349]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:25 smithi184 ceph-mon[100622]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:25 smithi184 ceph-mon[102844]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:27 smithi184 ceph-mon[100622]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:27 smithi184 ceph-mon[102844]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:27 smithi196 ceph-mon[76349]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:28.112 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 2h - - 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 29.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 13s ago 2h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:07:28.636 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 13s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 2h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 13s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:07:28.637 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:07:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 13s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:07:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:07:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:07:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:07:28.638 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:07:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:07:28.947 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:07:29.217 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:07:29.218 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:07:29.218 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:07:29.218 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:07:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:07:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[100622]: from='client.53181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[100622]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[100622]: from='client.63152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:29.218 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/978886191' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[102844]: from='client.53181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[102844]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[102844]: from='client.63152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:29.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/978886191' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:29.523 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:07:29.523 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:29.524 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:29.524 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:29.524 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:29 smithi196 ceph-mon[76349]: from='client.53181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:29 smithi196 ceph-mon[76349]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:29 smithi196 ceph-mon[76349]: from='client.63152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/978886191' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[102844]: from='client.63158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[102844]: from='client.53205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/640625083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[100622]: from='client.63158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[100622]: from='client.53205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/640625083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:30 smithi196 ceph-mon[76349]: from='client.63158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:30 smithi196 ceph-mon[76349]: from='client.53205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/640625083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:30.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:30.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:30.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:31 smithi184 ceph-mon[100622]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:31 smithi184 ceph-mon[102844]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:31 smithi196 ceph-mon[76349]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:07:33.479 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:33.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:33 smithi184 ceph-mon[100622]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:33.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:33.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:33 smithi184 ceph-mon[102844]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:33 smithi196 ceph-mon[76349]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:35 smithi196 ceph-mon[76349]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[102844]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:35 smithi184 ceph-mon[100622]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[100622]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:36 smithi184 ceph-mon[102844]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:36 smithi196 ceph-mon[76349]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:39.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:39 smithi184 ceph-mon[100622]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:39.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:39 smithi184 ceph-mon[102844]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:39 smithi196 ceph-mon[76349]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:40.911 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:40.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:40.911 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:40.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:40.911 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:40.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:41 smithi184 ceph-mon[100622]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:41 smithi184 ceph-mon[102844]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:41 smithi196 ceph-mon[76349]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:07:43.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:43.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:43 smithi196 ceph-mon[76349]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:43 smithi184 ceph-mon[100622]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:43 smithi184 ceph-mon[102844]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:45.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:45 smithi196 ceph-mon[76349]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:45 smithi184 ceph-mon[100622]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:45 smithi184 ceph-mon[102844]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:47.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:47 smithi196 ceph-mon[76349]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:47 smithi184 ceph-mon[100622]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:47 smithi184 ceph-mon[102844]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:49 smithi184 ceph-mon[100622]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:49 smithi184 ceph-mon[102844]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:49 smithi196 ceph-mon[76349]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:50.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:50.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:07:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:07:50.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:07:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:51 smithi184 ceph-mon[100622]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:51 smithi184 ceph-mon[102844]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:51 smithi196 ceph-mon[76349]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:07:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:07:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:53 smithi184 ceph-mon[100622]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:53 smithi184 ceph-mon[102844]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:53 smithi196 ceph-mon[76349]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[100622]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[102844]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:55 smithi196 ceph-mon[76349]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:56 smithi196 ceph-mon[76349]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[100622]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:07:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:56 smithi184 ceph-mon[102844]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:59 smithi184 ceph-mon[100622]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:59 smithi184 ceph-mon[102844]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:59 smithi196 ceph-mon[76349]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:59.821 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:08:00.385 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 2h - - 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 6s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 6s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 6s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 6s ago 2h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 4s ago 2h 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 6s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 6s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:08:00.386 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 4s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 4s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 4s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 4s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 4s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:08:00.387 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 4s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:08:00.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:08:00.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:08:00.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:08:00.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:08:00.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:08:01.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:00.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:01.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:00.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:01.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:00.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:01.017 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:08:01.017 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:08:01.017 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:08:01.017 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:08:01.017 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:08:01.018 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[100622]: from='client.53217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[100622]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[100622]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[100622]: from='client.63194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2062550374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[102844]: from='client.53217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[102844]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[102844]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[102844]: from='client.63194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2062550374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:01.336 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:08:01.336 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:01.337 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:01.337 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:01 smithi196 ceph-mon[76349]: from='client.53217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:01 smithi196 ceph-mon[76349]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:01 smithi196 ceph-mon[76349]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:01 smithi196 ceph-mon[76349]: from='client.63194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2062550374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:02.033 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:08:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:02 smithi184 ceph-mon[100622]: from='client.63206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/267319989' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:02 smithi184 ceph-mon[102844]: from='client.63206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/267319989' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:02 smithi196 ceph-mon[76349]: from='client.63206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/267319989' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:03 smithi184 ceph-mon[102844]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:03 smithi184 ceph-mon[100622]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:03 smithi196 ceph-mon[76349]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:05 smithi184 ceph-mon[100622]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:05 smithi184 ceph-mon[102844]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:05 smithi196 ceph-mon[76349]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:07 smithi196 ceph-mon[76349]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:07 smithi184 ceph-mon[100622]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:07 smithi184 ceph-mon[102844]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:09 smithi184 ceph-mon[100622]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:09 smithi184 ceph-mon[102844]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:09 smithi196 ceph-mon[76349]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:10.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:10.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:10.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:11 smithi184 ceph-mon[100622]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:11 smithi184 ceph-mon[102844]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:11 smithi196 ceph-mon[76349]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:08:13.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:13.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:13 smithi196 ceph-mon[76349]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:13 smithi184 ceph-mon[100622]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:13 smithi184 ceph-mon[102844]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:08:13.81252857Z level=info msg="Completed cleanup jobs" duration=12.790842ms 2024-09-20T22:08:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:15 smithi196 ceph-mon[76349]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[100622]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[102844]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[100622]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:16 smithi184 ceph-mon[102844]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:16 smithi196 ceph-mon[76349]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:19 smithi184 ceph-mon[100622]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:19 smithi184 ceph-mon[102844]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:19 smithi196 ceph-mon[76349]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:20.966 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:20.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:20.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:20.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:20.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:20.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:21.295 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:21 smithi184 ceph-mon[100622]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:21.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:21 smithi184 ceph-mon[102844]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:21 smithi196 ceph-mon[76349]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:08:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:23 smithi184 ceph-mon[100622]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:23 smithi184 ceph-mon[102844]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:23 smithi196 ceph-mon[76349]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:25 smithi184 ceph-mon[100622]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:25 smithi184 ceph-mon[102844]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:25 smithi196 ceph-mon[76349]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:27.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:27 smithi196 ceph-mon[76349]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:27 smithi184 ceph-mon[100622]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:27 smithi184 ceph-mon[102844]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:29 smithi184 ceph-mon[100622]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:29 smithi184 ceph-mon[102844]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:29 smithi196 ceph-mon[76349]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:30.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:30.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:30.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:31 smithi184 ceph-mon[100622]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:31 smithi184 ceph-mon[102844]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:31.630 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:08:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:31 smithi196 ceph-mon[76349]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 2h - - 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 16s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:08:32.157 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 16s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 2h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 16s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 16s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:08:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 16s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:08:32.159 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:08:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:08:32.473 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:08:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:08:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:08:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:08:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:08:32.475 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:08:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:08:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:08:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:08:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:08:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:08:32.746 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:08:33.065 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:08:33.065 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:33.065 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:33.065 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:33.066 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.53253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.53259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1246339254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3020971411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.53253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.53259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1246339254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3020971411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.53253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.53259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1246339254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3020971411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:34.464 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:34 smithi184 ceph-mon[100622]: from='client.53277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:34.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:34 smithi184 ceph-mon[102844]: from='client.53277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:34 smithi196 ceph-mon[76349]: from='client.53277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[100622]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[102844]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:35 smithi196 ceph-mon[76349]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:37 smithi196 ceph-mon[76349]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[100622]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:37 smithi184 ceph-mon[102844]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:39 smithi184 ceph-mon[100622]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:39 smithi184 ceph-mon[102844]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:39 smithi196 ceph-mon[76349]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:40.892 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:40.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:40.892 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:40.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:40.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:40.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:41 smithi184 ceph-mon[100622]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:41 smithi184 ceph-mon[102844]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:41 smithi196 ceph-mon[76349]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:08:43.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:43.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:43 smithi196 ceph-mon[76349]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:43 smithi184 ceph-mon[100622]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:43 smithi184 ceph-mon[102844]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:44.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:45 smithi184 ceph-mon[100622]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:45 smithi184 ceph-mon[102844]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:45 smithi196 ceph-mon[76349]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:47 smithi196 ceph-mon[76349]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:47 smithi184 ceph-mon[102844]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:47 smithi184 ceph-mon[100622]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.352 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:49 smithi196 ceph-mon[76349]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:49 smithi184 ceph-mon[100622]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:49 smithi184 ceph-mon[102844]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:50.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:50.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:08:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:08:50.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:08:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:51 smithi184 ceph-mon[100622]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:51 smithi184 ceph-mon[102844]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:51 smithi196 ceph-mon[76349]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:08:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:08:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:53 smithi184 ceph-mon[100622]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:53 smithi184 ceph-mon[102844]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:53 smithi196 ceph-mon[76349]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[102844]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[100622]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:55 smithi196 ceph-mon[76349]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[102844]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[100622]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:57 smithi196 ceph-mon[76349]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:08:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:59 smithi184 ceph-mon[100622]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:59 smithi184 ceph-mon[102844]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:59 smithi196 ceph-mon[76349]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:00.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:00.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:00.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:01.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:01 smithi184 ceph-mon[100622]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:01.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:01 smithi184 ceph-mon[102844]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:01 smithi196 ceph-mon[76349]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:09:03.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:03.365 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:03 smithi184 ceph-mon[100622]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:03.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:03.366 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:03 smithi184 ceph-mon[102844]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:03.366 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:09:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:03 smithi196 ceph-mon[76349]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:03.930 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:09:03.930 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 2h - - 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 9s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 9s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 9s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 9s ago 2h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 7s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 9s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 9s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 9s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 9s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:09:03.931 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 9s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 9s ago 2h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 7s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 7s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 7s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 7s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 9s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 9s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:09:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 7s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:09:04.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:09:04.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:09:04.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:09:04.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:04.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:09:04.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:09:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:04 smithi184 ceph-mon[100622]: from='client.63254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:04 smithi184 ceph-mon[102844]: from='client.63254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:04.558 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:09:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:09:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:09:04.559 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:09:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:04 smithi196 ceph-mon[76349]: from='client.63254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:04.868 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:09:04.868 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:04.868 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:04.868 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:04.868 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: from='client.63260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: from='client.63266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3035201698' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: from='client.53313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2157272928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: from='client.63260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: from='client.63266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3035201698' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: from='client.53313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2157272928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: from='client.63260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: from='client.63266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3035201698' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: from='client.53313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2157272928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:07.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:07 smithi196 ceph-mon[76349]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:07 smithi184 ceph-mon[102844]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:07 smithi184 ceph-mon[100622]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:09 smithi184 ceph-mon[100622]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:09 smithi184 ceph-mon[102844]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:09 smithi196 ceph-mon[76349]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:10.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:10.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:10.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:11 smithi184 ceph-mon[100622]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:11 smithi184 ceph-mon[102844]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:11 smithi196 ceph-mon[76349]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:09:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:13 smithi184 ceph-mon[100622]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:13 smithi184 ceph-mon[102844]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:13 smithi196 ceph-mon[76349]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:14.493 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:14.493 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[100622]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[102844]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:15 smithi196 ceph-mon[76349]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:15.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:15.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:17 smithi196 ceph-mon[76349]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[100622]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[102844]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:19 smithi196 ceph-mon[76349]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:19 smithi184 ceph-mon[100622]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:19 smithi184 ceph-mon[102844]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:20.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:20.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:20.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:21 smithi196 ceph-mon[76349]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:21 smithi184 ceph-mon[100622]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:21 smithi184 ceph-mon[102844]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:09:23.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:23.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:23 smithi184 ceph-mon[100622]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:23.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:23 smithi184 ceph-mon[102844]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:23 smithi196 ceph-mon[76349]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:25 smithi196 ceph-mon[76349]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:25 smithi184 ceph-mon[100622]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:25 smithi184 ceph-mon[102844]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:27 smithi196 ceph-mon[76349]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:27 smithi184 ceph-mon[100622]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:27 smithi184 ceph-mon[102844]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:29 smithi196 ceph-mon[76349]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:29 smithi184 ceph-mon[100622]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:29 smithi184 ceph-mon[102844]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:30.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:30.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:30.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:31 smithi196 ceph-mon[76349]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:31 smithi184 ceph-mon[100622]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:31 smithi184 ceph-mon[102844]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:09:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:33 smithi196 ceph-mon[76349]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:33.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:33.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:33 smithi184 ceph-mon[100622]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:33.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:33.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:33 smithi184 ceph-mon[102844]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:35.151 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[100622]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[102844]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:35 smithi196 ceph-mon[76349]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 2h - - 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 2h - - 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 0s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 19s ago 2h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:09:35.685 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 0s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 19s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 0s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 19s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:09:35.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:09:35.687 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:09:35.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:09:35.996 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:09:36.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:09:36.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:09:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[100622]: from='client.53325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[100622]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2649930015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[102844]: from='client.53325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[102844]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2649930015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:36.580 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:09:36.580 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:36.580 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:36.580 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:36.581 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:36 smithi196 ceph-mon[76349]: from='client.53325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:36 smithi196 ceph-mon[76349]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2649930015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='client.63302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='client.63314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1757948179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='client.63302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='client.63314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1757948179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='client.63302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='client.63314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1757948179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:39 smithi184 ceph-mon[100622]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:39 smithi184 ceph-mon[102844]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:39 smithi196 ceph-mon[76349]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:40.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:40.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:40.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:41 smithi184 ceph-mon[100622]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:41 smithi184 ceph-mon[102844]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:41 smithi196 ceph-mon[76349]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:09:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:43 smithi184 ceph-mon[100622]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:43 smithi184 ceph-mon[102844]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:43 smithi196 ceph-mon[76349]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:44.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:44.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:45 smithi184 ceph-mon[100622]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:45 smithi184 ceph-mon[102844]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:45 smithi196 ceph-mon[76349]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:47 smithi184 ceph-mon[102844]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:47 smithi184 ceph-mon[100622]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:47 smithi196 ceph-mon[76349]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:49.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:49 smithi196 ceph-mon[76349]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:49 smithi184 ceph-mon[102844]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:49 smithi184 ceph-mon[100622]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:50.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:50.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:09:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:09:50.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:09:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:51 smithi184 ceph-mon[102844]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:51 smithi184 ceph-mon[100622]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:51 smithi196 ceph-mon[76349]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:09:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:09:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:53 smithi184 ceph-mon[100622]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:53 smithi184 ceph-mon[102844]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:53 smithi196 ceph-mon[76349]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[100622]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[102844]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:55 smithi196 ceph-mon[76349]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[100622]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[102844]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:57 smithi196 ceph-mon[76349]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:09:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:59 smithi184 ceph-mon[100622]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:59 smithi184 ceph-mon[102844]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:59 smithi196 ceph-mon[76349]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:00.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:00.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:00.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:00.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:00.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:01 smithi184 ceph-mon[100622]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:01 smithi184 ceph-mon[102844]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:01 smithi196 ceph-mon[76349]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:03 smithi184 ceph-mon[102844]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:03 smithi184 ceph-mon[100622]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:03 smithi196 ceph-mon[76349]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:05.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:05 smithi184 ceph-mon[100622]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:05.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:05 smithi184 ceph-mon[102844]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:05 smithi196 ceph-mon[76349]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:06.870 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 93.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:10:07.400 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 12s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 12s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 12s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 12s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 12s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:10:07.401 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:10:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 10s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:10:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 12s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:10:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:10:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 12s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:10:07.402 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:10:07.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:10:07.708 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:10:07.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:10:07.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:07 smithi184 ceph-mon[100622]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:07.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:07 smithi184 ceph-mon[102844]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:07 smithi196 ceph-mon[76349]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:10:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:10:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:10:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:10:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:10:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:10:07.980 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:10:08.291 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:08.291 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:08.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:08.291 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:08.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[100622]: from='client.53361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[100622]: from='client.63332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[100622]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3653718452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3486088440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[102844]: from='client.53361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[102844]: from='client.63332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[102844]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3653718452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:08.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3486088440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:08 smithi196 ceph-mon[76349]: from='client.53361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:08 smithi196 ceph-mon[76349]: from='client.63332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:08 smithi196 ceph-mon[76349]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3653718452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3486088440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:09 smithi184 ceph-mon[100622]: from='client.53385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:09 smithi184 ceph-mon[100622]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:09 smithi184 ceph-mon[102844]: from='client.53385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:09 smithi184 ceph-mon[102844]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:09 smithi196 ceph-mon[76349]: from='client.53385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:09 smithi196 ceph-mon[76349]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:10.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:10.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:10.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:11 smithi196 ceph-mon[76349]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:11 smithi184 ceph-mon[100622]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:11 smithi184 ceph-mon[102844]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:10:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:13 smithi184 ceph-mon[102844]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:13 smithi184 ceph-mon[100622]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:13 smithi196 ceph-mon[76349]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[102844]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[100622]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:15 smithi196 ceph-mon[76349]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[102844]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[100622]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:17 smithi196 ceph-mon[76349]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:19 smithi184 ceph-mon[100622]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:19 smithi184 ceph-mon[102844]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:19 smithi196 ceph-mon[76349]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:20.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:20.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:20.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:21 smithi184 ceph-mon[100622]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:21 smithi184 ceph-mon[102844]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:21 smithi196 ceph-mon[76349]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:10:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:23 smithi184 ceph-mon[100622]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:23 smithi184 ceph-mon[102844]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:23 smithi196 ceph-mon[76349]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:25 smithi184 ceph-mon[100622]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:25 smithi184 ceph-mon[102844]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:25 smithi196 ceph-mon[76349]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:27 smithi184 ceph-mon[100622]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:27 smithi184 ceph-mon[102844]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:27 smithi196 ceph-mon[76349]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:29.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:29 smithi196 ceph-mon[76349]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:29.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:29 smithi184 ceph-mon[100622]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:29 smithi184 ceph-mon[102844]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:30.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:30.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:30.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:31 smithi196 ceph-mon[76349]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:31 smithi184 ceph-mon[100622]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:31 smithi184 ceph-mon[102844]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:10:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:33 smithi184 ceph-mon[100622]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:33 smithi184 ceph-mon[102844]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:33 smithi196 ceph-mon[76349]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:35 smithi196 ceph-mon[76349]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[100622]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[102844]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:37 smithi196 ceph-mon[76349]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[100622]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[102844]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:38.594 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 2h - - 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 2h - - 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 3s ago 2h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 2s ago 2h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 3s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 2s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 3s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 2h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:10:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 3s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 2s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 2s ago 2h 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:10:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:10:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:10:39.433 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:10:39.701 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:10:39.702 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:10:39.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[100622]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:39.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[100622]: from='client.63362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:39.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3720745385' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:39.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[102844]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:39.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[102844]: from='client.63362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:39.703 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3720745385' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:39 smithi196 ceph-mon[76349]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:39 smithi196 ceph-mon[76349]: from='client.63362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3720745385' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:40.012 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:40.012 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:40.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:40.012 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:40.013 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:40 smithi196 ceph-mon[76349]: from='client.63368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:40 smithi196 ceph-mon[76349]: from='client.53409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2680473437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[100622]: from='client.63368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[100622]: from='client.53409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2680473437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[102844]: from='client.63368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[102844]: from='client.53409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2680473437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:40.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:40.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:40.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:41 smithi196 ceph-mon[76349]: from='client.63386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:41 smithi196 ceph-mon[76349]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:41 smithi184 ceph-mon[100622]: from='client.63386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:41 smithi184 ceph-mon[100622]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:41 smithi184 ceph-mon[102844]: from='client.63386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:41 smithi184 ceph-mon[102844]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:10:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:43 smithi184 ceph-mon[100622]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:43 smithi184 ceph-mon[102844]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:43 smithi196 ceph-mon[76349]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:44.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:45.664 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:45 smithi184 ceph-mon[100622]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:45.664 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:45 smithi184 ceph-mon[102844]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:45 smithi196 ceph-mon[76349]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:47 smithi196 ceph-mon[76349]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:47 smithi184 ceph-mon[100622]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:47 smithi184 ceph-mon[102844]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:49 smithi196 ceph-mon[76349]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:49 smithi184 ceph-mon[100622]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:49 smithi184 ceph-mon[102844]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:50 smithi196 ceph-mon[76349]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:50 smithi184 ceph-mon[100622]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:50 smithi184 ceph-mon[102844]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:50.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:50.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:10:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:10:50.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:10:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:10:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:53 smithi184 ceph-mon[100622]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:53 smithi184 ceph-mon[102844]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:53 smithi196 ceph-mon[76349]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:55 smithi184 ceph-mon[100622]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:55 smithi184 ceph-mon[102844]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:55 smithi196 ceph-mon[76349]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:57 smithi196 ceph-mon[76349]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[100622]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[102844]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:10:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:59 smithi196 ceph-mon[76349]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:10:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:59 smithi184 ceph-mon[100622]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:10:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:59 smithi184 ceph-mon[102844]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:10:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:00.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:00.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:00.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:01 smithi196 ceph-mon[76349]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:01 smithi184 ceph-mon[100622]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:01 smithi184 ceph-mon[102844]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:11:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:03 smithi196 ceph-mon[76349]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:03 smithi184 ceph-mon[100622]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:03 smithi184 ceph-mon[102844]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:05 smithi196 ceph-mon[76349]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:05 smithi184 ceph-mon[100622]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:05 smithi184 ceph-mon[102844]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:07.497 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:07 smithi184 ceph-mon[100622]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:07.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:07 smithi184 ceph-mon[102844]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:07 smithi196 ceph-mon[76349]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:09 smithi196 ceph-mon[76349]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:09 smithi184 ceph-mon[100622]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:09 smithi184 ceph-mon[102844]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:10.309 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 2h - - 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 13s ago 2h 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:11:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 13s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 15s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 15s ago 2h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 13s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 15s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 15s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 15s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 15s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:11:10.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 13s ago 2h 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 15s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:11:10.840 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:11:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:10.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:10.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:10.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:11:11.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:11:11.148 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:11:11.420 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:11:11.420 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:11:11.420 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:11:11.420 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:11:11.420 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:11:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:11:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[102844]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:11.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[102844]: from='client.53433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[102844]: from='client.63404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3371536316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[100622]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[100622]: from='client.53433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[100622]: from='client.63404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3371536316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:11 smithi196 ceph-mon[76349]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:11 smithi196 ceph-mon[76349]: from='client.53433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:11 smithi196 ceph-mon[76349]: from='client.63404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3371536316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:11:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:12 smithi196 ceph-mon[76349]: from='client.63410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:12 smithi196 ceph-mon[76349]: from='client.63422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3737197666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[100622]: from='client.63410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[100622]: from='client.63422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3737197666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[102844]: from='client.63410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[102844]: from='client.63422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3737197666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:13 smithi196 ceph-mon[76349]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:13 smithi184 ceph-mon[100622]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:13 smithi184 ceph-mon[102844]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:14.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:15 smithi184 ceph-mon[100622]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:15 smithi184 ceph-mon[102844]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:15.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:15 smithi196 ceph-mon[76349]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[100622]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[102844]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:17 smithi196 ceph-mon[76349]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:19 smithi184 ceph-mon[100622]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:19 smithi184 ceph-mon[102844]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:19 smithi196 ceph-mon[76349]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:20.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:20.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:20.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:21 smithi184 ceph-mon[100622]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:21 smithi184 ceph-mon[102844]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:21 smithi196 ceph-mon[76349]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:11:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:23 smithi184 ceph-mon[100622]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:23 smithi184 ceph-mon[102844]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:23 smithi196 ceph-mon[76349]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:25 smithi196 ceph-mon[76349]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:25 smithi184 ceph-mon[102844]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:25 smithi184 ceph-mon[100622]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:27.740 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:27 smithi184 ceph-mon[100622]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:27.740 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:27 smithi184 ceph-mon[102844]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:27 smithi196 ceph-mon[76349]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:29 smithi184 ceph-mon[100622]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:29 smithi184 ceph-mon[102844]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:29 smithi196 ceph-mon[76349]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:30.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:30.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:30.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:31.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:31 smithi196 ceph-mon[76349]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:31 smithi184 ceph-mon[100622]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:31 smithi184 ceph-mon[102844]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:11:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:33 smithi196 ceph-mon[76349]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:33 smithi184 ceph-mon[100622]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:33.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:33.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:33 smithi184 ceph-mon[102844]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:33.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:35 smithi184 ceph-mon[100622]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:35 smithi184 ceph-mon[102844]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:35 smithi196 ceph-mon[76349]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:37 smithi196 ceph-mon[76349]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[100622]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[102844]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:38 smithi196 ceph-mon[76349]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:38 smithi184 ceph-mon[100622]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:38 smithi184 ceph-mon[102844]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:40.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:40.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:40.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:41 smithi184 ceph-mon[100622]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:41 smithi184 ceph-mon[102844]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:41 smithi196 ceph-mon[76349]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:42.023 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:11:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:11:42.551 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:11:42.551 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 2h - - 2024-09-20T22:11:42.551 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T22:11:42.551 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 6s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 6s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 6s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 6s ago 2h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 6s ago 2h 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 6s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:11:42.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 75.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:11:42.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:11:42.860 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:11:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:11:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:11:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:11:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:11:42.862 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:11:43.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:11:43.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:11:43.444 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:11:43.445 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:43.445 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:43.445 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:43.445 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: from='client.53469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: from='client.63440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: from='client.63446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1986566221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: from='client.53469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:43.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: from='client.63440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: from='client.63446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1986566221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: from='client.53469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: from='client.63440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: from='client.63446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1986566221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:44 smithi196 ceph-mon[76349]: from='client.63458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2637278227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:44.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[100622]: from='client.63458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:44.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2637278227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:44.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:44.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[102844]: from='client.63458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:44.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2637278227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:44.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:45 smithi184 ceph-mon[100622]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:45 smithi184 ceph-mon[102844]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:45 smithi196 ceph-mon[76349]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:47 smithi184 ceph-mon[100622]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:47 smithi184 ceph-mon[102844]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:47 smithi196 ceph-mon[76349]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:49.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:49 smithi184 ceph-mon[100622]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:49.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:49 smithi184 ceph-mon[102844]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:49 smithi196 ceph-mon[76349]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:50.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:50.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:11:50.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:11:50.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:11:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:51 smithi184 ceph-mon[100622]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:51 smithi184 ceph-mon[102844]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:51 smithi196 ceph-mon[76349]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:11:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:11:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:53 smithi184 ceph-mon[100622]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:53 smithi184 ceph-mon[102844]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:53 smithi196 ceph-mon[76349]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:55 smithi184 ceph-mon[100622]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:55 smithi184 ceph-mon[102844]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:55 smithi196 ceph-mon[76349]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[100622]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:56 smithi184 ceph-mon[102844]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:56 smithi196 ceph-mon[76349]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:11:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:59 smithi196 ceph-mon[76349]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:59 smithi184 ceph-mon[100622]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:59 smithi184 ceph-mon[102844]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:00.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:00.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:00.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:01 smithi196 ceph-mon[76349]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:01 smithi184 ceph-mon[100622]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:01 smithi184 ceph-mon[102844]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:12:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:03 smithi196 ceph-mon[76349]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:03.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:03 smithi184 ceph-mon[100622]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:03.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:03.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:03 smithi184 ceph-mon[102844]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:03.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:05 smithi184 ceph-mon[100622]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:05 smithi184 ceph-mon[102844]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:05 smithi196 ceph-mon[76349]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:07 smithi184 ceph-mon[100622]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:07 smithi184 ceph-mon[102844]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:07 smithi196 ceph-mon[76349]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:09 smithi184 ceph-mon[100622]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:09 smithi184 ceph-mon[102844]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:09 smithi196 ceph-mon[76349]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:10.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:10.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:10.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:11 smithi184 ceph-mon[100622]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:11 smithi184 ceph-mon[102844]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:11 smithi196 ceph-mon[76349]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:12.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:12:13.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:13 smithi184 ceph-mon[100622]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:13.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:13 smithi184 ceph-mon[102844]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:13.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:13 smithi196 ceph-mon[76349]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:13.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:13.745 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 2h - - 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 2h - - 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 16s ago 2h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:12:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 16s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 18s ago 2h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 16s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 18s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 16s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 18s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 18s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 18s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:12:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 16s ago 2h 75.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 18s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:12:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:12:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:12:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:12:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:12:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:12:14.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:12:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.850 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:12:14.850 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:12:14.850 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:12:14.850 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:12:14.851 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:12:15.159 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:12:15.159 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:15.159 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:15.159 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:15.159 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: from='client.53505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: from='client.63482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2874040346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:15.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3666149777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: from='client.53505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: from='client.63482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2874040346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3666149777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: from='client.53505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: from='client.63482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2874040346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3666149777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[100622]: from='client.63494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[102844]: from='client.63494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:16 smithi196 ceph-mon[76349]: from='client.63494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:17 smithi184 ceph-mon[100622]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:17 smithi184 ceph-mon[102844]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:17 smithi196 ceph-mon[76349]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:19 smithi184 ceph-mon[100622]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:19 smithi184 ceph-mon[102844]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:19 smithi196 ceph-mon[76349]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:20.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:20.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:20.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:21 smithi184 ceph-mon[100622]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:21 smithi184 ceph-mon[102844]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:21 smithi196 ceph-mon[76349]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:22.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:12:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:23 smithi184 ceph-mon[102844]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:23 smithi184 ceph-mon[100622]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:23 smithi196 ceph-mon[76349]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:25 smithi184 ceph-mon[100622]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:25 smithi184 ceph-mon[102844]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:25 smithi196 ceph-mon[76349]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:27 smithi184 ceph-mon[100622]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:27 smithi184 ceph-mon[102844]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:27 smithi196 ceph-mon[76349]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:29 smithi184 ceph-mon[100622]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:29 smithi184 ceph-mon[102844]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:29 smithi196 ceph-mon[76349]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:30.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:30.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:30.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:31.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:31 smithi184 ceph-mon[100622]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:31.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:31 smithi184 ceph-mon[102844]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:31 smithi196 ceph-mon[76349]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:12:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:33 smithi184 ceph-mon[100622]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:33 smithi184 ceph-mon[102844]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:33.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:33 smithi196 ceph-mon[76349]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:33.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:35 smithi184 ceph-mon[100622]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:35 smithi184 ceph-mon[102844]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:35.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:35 smithi196 ceph-mon[76349]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[100622]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:36 smithi184 ceph-mon[102844]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:36 smithi196 ceph-mon[76349]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:38 smithi196 ceph-mon[76349]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[100622]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:38 smithi184 ceph-mon[102844]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:40.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:40.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:40.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:41 smithi184 ceph-mon[100622]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:41 smithi184 ceph-mon[102844]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:41 smithi196 ceph-mon[76349]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:42.279 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:12:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:43 smithi184 ceph-mon[100622]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:43 smithi184 ceph-mon[102844]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:43 smithi196 ceph-mon[76349]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:44.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:45.457 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:12:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:45 smithi184 ceph-mon[100622]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:45 smithi184 ceph-mon[102844]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:45 smithi196 ceph-mon[76349]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:45.980 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 2h - - 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 2h - - 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 8s ago 2h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 2h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 8s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 8s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:12:45.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 10s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 10s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 8s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 8s ago 2h 76.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:12:45.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:12:46.296 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:12:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:12:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:12:46.298 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:12:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:46 smithi184 ceph-mon[100622]: from='client.63506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:46 smithi184 ceph-mon[102844]: from='client.63506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:12:46.566 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:12:46.567 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:12:46.567 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:12:46.567 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:12:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:46 smithi196 ceph-mon[76349]: from='client.63506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:46.874 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:12:46.875 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:46.875 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:46.875 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:46.875 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: from='client.63512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2290503238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: from='client.63530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1241243137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: from='client.63512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2290503238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: from='client.63530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1241243137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: from='client.63512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2290503238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: from='client.63530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1241243137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:49 smithi184 ceph-mon[100622]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:49 smithi184 ceph-mon[102844]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:49 smithi196 ceph-mon[76349]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:51.020 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:50.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:51.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:50.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:12:51.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:12:50.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:12:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:51 smithi184 ceph-mon[100622]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:51 smithi184 ceph-mon[102844]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:51 smithi196 ceph-mon[76349]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:52.287 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:12:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:12:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:53 smithi184 ceph-mon[100622]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:53 smithi184 ceph-mon[102844]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:53 smithi196 ceph-mon[76349]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:55.456 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:55 smithi196 ceph-mon[76349]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:55 smithi184 ceph-mon[100622]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:55 smithi184 ceph-mon[102844]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:57 smithi196 ceph-mon[76349]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[100622]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:57 smithi184 ceph-mon[102844]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[100622]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:58 smithi184 ceph-mon[102844]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:12:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:58 smithi196 ceph-mon[76349]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:00 smithi184 ceph-mon[100622]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:00 smithi184 ceph-mon[102844]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:00.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:00.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:00.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:00 smithi196 ceph-mon[76349]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:02.293 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:03 smithi184 ceph-mon[100622]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:03 smithi184 ceph-mon[102844]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:03 smithi196 ceph-mon[76349]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:05 smithi184 ceph-mon[100622]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:05 smithi184 ceph-mon[102844]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:05 smithi196 ceph-mon[76349]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:07 smithi184 ceph-mon[100622]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:07 smithi184 ceph-mon[102844]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:07 smithi196 ceph-mon[76349]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:09 smithi184 ceph-mon[100622]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:09 smithi184 ceph-mon[102844]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:09 smithi196 ceph-mon[76349]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:10.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:10.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:10.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:11.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:11 smithi184 ceph-mon[100622]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:11.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:11 smithi184 ceph-mon[102844]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:11 smithi196 ceph-mon[76349]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:12.131 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:13:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:13 smithi184 ceph-mon[102844]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:13 smithi184 ceph-mon[100622]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:13 smithi196 ceph-mon[76349]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:15 smithi184 ceph-mon[100622]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:15 smithi184 ceph-mon[102844]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:15.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:15 smithi196 ceph-mon[76349]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:17.172 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:13:17.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:17 smithi196 ceph-mon[76349]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:17.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[100622]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[102844]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 2h - - 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 19s ago 2h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 2h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:13:17.698 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 19s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 2h 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 19s ago 2h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 2h 76.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:13:17.699 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:13:17.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:13:17.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:13:18.013 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:13:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:13:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:13:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:13:18.014 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:13:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:13:18.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:13:18.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='client.63542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='client.53583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2362150535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='client.63542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='client.53583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2362150535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:18.595 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:13:18.595 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:18.595 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:18.595 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:18.595 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='client.63542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='client.53583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2362150535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[100622]: from='client.63554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[100622]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[100622]: from='client.53601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2023938827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[102844]: from='client.63554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[102844]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[102844]: from='client.53601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2023938827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:19 smithi196 ceph-mon[76349]: from='client.63554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:19 smithi196 ceph-mon[76349]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:19 smithi196 ceph-mon[76349]: from='client.53601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2023938827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:20.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:20.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:20.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:21 smithi184 ceph-mon[100622]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:21 smithi184 ceph-mon[102844]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:21 smithi196 ceph-mon[76349]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:13:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:23 smithi196 ceph-mon[76349]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:23 smithi184 ceph-mon[100622]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:23 smithi184 ceph-mon[102844]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:25 smithi184 ceph-mon[100622]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:25 smithi184 ceph-mon[102844]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:25 smithi196 ceph-mon[76349]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:27 smithi184 ceph-mon[100622]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:27 smithi184 ceph-mon[102844]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:27 smithi196 ceph-mon[76349]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:29 smithi184 ceph-mon[100622]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:29 smithi184 ceph-mon[102844]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:29 smithi196 ceph-mon[76349]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:30.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:30.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:30.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:31.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:31 smithi184 ceph-mon[100622]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:31.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:31 smithi184 ceph-mon[102844]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:31 smithi196 ceph-mon[76349]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:32.216 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:13:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:33 smithi184 ceph-mon[100622]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:33 smithi184 ceph-mon[102844]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:33 smithi196 ceph-mon[76349]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:35 smithi196 ceph-mon[76349]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:35 smithi184 ceph-mon[100622]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:35 smithi184 ceph-mon[102844]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:37 smithi196 ceph-mon[76349]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[100622]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[102844]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:37.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[100622]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:38 smithi184 ceph-mon[102844]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:38 smithi196 ceph-mon[76349]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:40.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:40.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:40.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:41 smithi184 ceph-mon[100622]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:41 smithi184 ceph-mon[102844]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:41 smithi196 ceph-mon[76349]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:13:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:43 smithi184 ceph-mon[102844]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:43 smithi184 ceph-mon[100622]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:43 smithi196 ceph-mon[76349]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:13:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:44.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:45 smithi196 ceph-mon[76349]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:45 smithi184 ceph-mon[100622]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:45 smithi184 ceph-mon[102844]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:47 smithi196 ceph-mon[76349]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:47 smithi184 ceph-mon[100622]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:47 smithi184 ceph-mon[102844]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:48.887 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:13:49.414 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:13:49.414 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 2h - - 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 2h - - 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 13s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 13s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 13s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 13s ago 2h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 11s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 13s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 13s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 13s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 13s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 2h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:13:49.415 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 11s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 11s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 2h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 76.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:13:49.416 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:13:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:49 smithi196 ceph-mon[76349]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:49.727 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:13:49.728 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:13:49.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:49 smithi184 ceph-mon[100622]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:49.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:49 smithi184 ceph-mon[102844]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:13:49.998 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:13:49.999 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:13:49.999 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:13:49.999 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:13:50.308 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:13:50.308 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:50.308 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:50.308 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:50.308 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:50 smithi196 ceph-mon[76349]: from='client.53613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:50 smithi196 ceph-mon[76349]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:50 smithi196 ceph-mon[76349]: from='client.63590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3242844264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[100622]: from='client.53613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[100622]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[100622]: from='client.63590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3242844264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[102844]: from='client.53613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[102844]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[102844]: from='client.63590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:50.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3242844264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:50.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:50.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:13:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:13:50.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:13:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:51 smithi196 ceph-mon[76349]: from='client.63602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:51 smithi196 ceph-mon[76349]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3006202689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[100622]: from='client.63602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[100622]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3006202689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[102844]: from='client.63602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[102844]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3006202689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:52.147 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:13:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:13:53.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:53 smithi184 ceph-mon[100622]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:53.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:53.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:53 smithi184 ceph-mon[102844]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:53.496 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:53 smithi196 ceph-mon[76349]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:55 smithi184 ceph-mon[100622]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:55 smithi184 ceph-mon[102844]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:55 smithi196 ceph-mon[76349]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:57 smithi196 ceph-mon[76349]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[100622]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[102844]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:59 smithi196 ceph-mon[76349]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[102844]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[100622]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:00.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:00.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:00.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:01 smithi184 ceph-mon[100622]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:01 smithi184 ceph-mon[102844]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:01 smithi196 ceph-mon[76349]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:14:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:03 smithi184 ceph-mon[100622]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:03 smithi184 ceph-mon[102844]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:03 smithi196 ceph-mon[76349]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:05 smithi184 ceph-mon[100622]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:14:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:05 smithi184 ceph-mon[102844]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:14:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:05 smithi196 ceph-mon[76349]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:14:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:07 smithi196 ceph-mon[76349]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:14:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:07 smithi184 ceph-mon[100622]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:14:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:07 smithi184 ceph-mon[102844]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:14:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:09 smithi196 ceph-mon[76349]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T22:14:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:09 smithi184 ceph-mon[100622]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T22:14:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:09 smithi184 ceph-mon[102844]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T22:14:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:10.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:10.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:10.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:11 smithi196 ceph-mon[76349]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-20T22:14:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:11 smithi184 ceph-mon[102844]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-20T22:14:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:11 smithi184 ceph-mon[100622]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-20T22:14:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:14:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:13 smithi184 ceph-mon[100622]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:14:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:13 smithi184 ceph-mon[102844]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:14:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:13 smithi196 ceph-mon[76349]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:14:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:14.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:15.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:15 smithi184 ceph-mon[100622]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:14:15.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:15 smithi184 ceph-mon[102844]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:14:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:15 smithi196 ceph-mon[76349]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:14:17.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:17 smithi196 ceph-mon[76349]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:14:17.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:17.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:17.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[100622]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[102844]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:19 smithi196 ceph-mon[76349]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T22:14:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[102844]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T22:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:19 smithi184 ceph-mon[100622]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-20T22:14:20.621 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:14:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:20.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:20.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:20.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 2h - - 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 2h - - 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 2s ago 2h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 2s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:14:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 2h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 2s ago 2h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 4s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 4s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 4s ago 2h 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 2h 83.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 2h 71.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 2s ago 2h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:14:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:14:21.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:14:21.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 4s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:14:21.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:14:21.484 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:14:21.484 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:14:21.485 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:14:21.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:21 smithi184 ceph-mon[100622]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:21.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:21 smithi184 ceph-mon[100622]: from='client.63614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:21.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:21 smithi184 ceph-mon[102844]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:21.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:21 smithi184 ceph-mon[102844]: from='client.63614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:21 smithi196 ceph-mon[76349]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:21 smithi196 ceph-mon[76349]: from='client.63614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:21.753 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:14:21.754 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:14:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:14:22.061 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:14:22.061 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:22.061 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:22.062 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:22.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[100622]: from='client.53655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[100622]: from='client.53661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1000680673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1667162385' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[102844]: from='client.53655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[102844]: from='client.53661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1000680673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:22.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1667162385' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:22 smithi196 ceph-mon[76349]: from='client.53655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:22 smithi196 ceph-mon[76349]: from='client.53661 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1000680673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1667162385' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:23 smithi196 ceph-mon[76349]: from='client.63638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:23 smithi196 ceph-mon[76349]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:14:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[100622]: from='client.63638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[100622]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:14:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[102844]: from='client.63638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[102844]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:14:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:25 smithi196 ceph-mon[76349]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:25 smithi184 ceph-mon[100622]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:25 smithi184 ceph-mon[102844]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:14:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:27 smithi196 ceph-mon[76349]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:14:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:27 smithi184 ceph-mon[100622]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:14:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:27 smithi184 ceph-mon[102844]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:14:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:29 smithi196 ceph-mon[76349]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:14:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:29 smithi184 ceph-mon[100622]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:14:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:29 smithi184 ceph-mon[102844]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:14:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:30.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:30.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:30.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:31 smithi196 ceph-mon[76349]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:14:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:31 smithi184 ceph-mon[100622]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:14:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:31 smithi184 ceph-mon[102844]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:14:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:32] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T22:14:33.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:33 smithi184 ceph-mon[100622]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:14:33.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:33.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:33 smithi184 ceph-mon[102844]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:14:33.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:33 smithi196 ceph-mon[76349]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:14:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:35 smithi196 ceph-mon[76349]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:14:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:35 smithi184 ceph-mon[100622]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:14:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:35 smithi184 ceph-mon[102844]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:14:37.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:37 smithi196 ceph-mon[76349]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T22:14:37.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[100622]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[102844]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:39 smithi196 ceph-mon[76349]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:14:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[100622]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[102844]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:40.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:40.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:40.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:41 smithi196 ceph-mon[76349]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:14:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:41 smithi184 ceph-mon[100622]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:14:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:41 smithi184 ceph-mon[102844]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:14:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:42] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T22:14:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:43 smithi196 ceph-mon[76349]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:14:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:43 smithi184 ceph-mon[100622]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:14:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:43 smithi184 ceph-mon[102844]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:14:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:45 smithi196 ceph-mon[76349]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:45 smithi184 ceph-mon[100622]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:45 smithi184 ceph-mon[102844]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:47 smithi196 ceph-mon[76349]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:47 smithi184 ceph-mon[100622]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:47 smithi184 ceph-mon[102844]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:49 smithi196 ceph-mon[76349]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:49 smithi184 ceph-mon[100622]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:49 smithi184 ceph-mon[102844]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:50.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:50.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:14:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:14:50.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:14:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:51 smithi196 ceph-mon[76349]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:51 smithi184 ceph-mon[100622]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:51 smithi184 ceph-mon[102844]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:14:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:14:52.360 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:14:52.892 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:14:52.892 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T22:14:52.892 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T22:14:52.892 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 16s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:14:52.892 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 16s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 16s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 16s ago 2h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 16s ago 2h 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 16s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 16s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 16s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 16s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 16s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 2h 71.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:14:52.893 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:14:52.894 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:14:53.205 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:14:53.205 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:14:53.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:14:53.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:14:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:14:53.474 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:14:53.474 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:14:53.474 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:14:53.474 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:14:53.475 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:14:53.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[100622]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:53.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[100622]: from='client.53685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[100622]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/369598467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[102844]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[102844]: from='client.53685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[102844]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/369598467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:53 smithi196 ceph-mon[76349]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:53 smithi196 ceph-mon[76349]: from='client.53685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:53 smithi196 ceph-mon[76349]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/369598467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:53.783 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:14:53.784 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:53.784 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:53.784 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:53.784 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[100622]: from='client.53697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[100622]: from='client.63674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2416501688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[102844]: from='client.53697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[102844]: from='client.63674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2416501688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:54 smithi196 ceph-mon[76349]: from='client.53697 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:54 smithi196 ceph-mon[76349]: from='client.63674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2416501688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:55.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:55 smithi184 ceph-mon[100622]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:55.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:55 smithi184 ceph-mon[102844]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:55 smithi196 ceph-mon[76349]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:57 smithi196 ceph-mon[76349]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[100622]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[102844]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:59 smithi196 ceph-mon[76349]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[100622]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[102844]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:00.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:00.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:00.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:01 smithi196 ceph-mon[76349]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:01 smithi184 ceph-mon[100622]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:01 smithi184 ceph-mon[102844]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:15:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:03 smithi184 ceph-mon[100622]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:03 smithi184 ceph-mon[102844]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:03 smithi196 ceph-mon[76349]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:05 smithi184 ceph-mon[100622]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:05 smithi184 ceph-mon[102844]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:05 smithi196 ceph-mon[76349]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:07 smithi184 ceph-mon[100622]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:07 smithi184 ceph-mon[102844]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:07 smithi196 ceph-mon[76349]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:09 smithi184 ceph-mon[100622]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:09 smithi184 ceph-mon[102844]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:09 smithi196 ceph-mon[76349]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:10.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:10.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:10.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:11 smithi196 ceph-mon[76349]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:11 smithi184 ceph-mon[100622]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:11 smithi184 ceph-mon[102844]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:15:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:13 smithi184 ceph-mon[100622]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:13 smithi184 ceph-mon[102844]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:13 smithi196 ceph-mon[76349]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:15 smithi184 ceph-mon[100622]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:15 smithi184 ceph-mon[102844]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:15 smithi196 ceph-mon[76349]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:17.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[100622]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:17.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[102844]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:17.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:17 smithi196 ceph-mon[76349]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:19 smithi196 ceph-mon[76349]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[100622]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[102844]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:20.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:20.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:20.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:21 smithi196 ceph-mon[76349]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:21 smithi184 ceph-mon[100622]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:21 smithi184 ceph-mon[102844]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:15:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:23 smithi196 ceph-mon[76349]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:23 smithi184 ceph-mon[100622]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:23 smithi184 ceph-mon[102844]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:24.081 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 2h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 2h 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:15:24.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 7s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 7s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 7s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 7s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:15:24.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:15:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:15:24.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:15:25.188 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:15:25.231 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:15:25.231 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:15:25.231 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:15:25.231 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:15:25.232 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:15:25.497 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:15:25.498 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:25.498 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:25.498 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:25.498 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: from='client.63686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: from='client.63692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: from='client.63698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3601216887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/388674023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: from='client.63686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: from='client.63692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: from='client.63698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3601216887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/388674023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: from='client.63686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: from='client.63692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: from='client.63698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3601216887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/388674023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:26 smithi196 ceph-mon[76349]: from='client.53745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:26 smithi184 ceph-mon[100622]: from='client.53745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:26 smithi184 ceph-mon[102844]: from='client.53745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:27 smithi196 ceph-mon[76349]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:27 smithi184 ceph-mon[100622]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:27 smithi184 ceph-mon[102844]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:29 smithi196 ceph-mon[76349]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:29 smithi184 ceph-mon[100622]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:29 smithi184 ceph-mon[102844]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:30.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:30.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:30.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:31 smithi196 ceph-mon[76349]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:31 smithi184 ceph-mon[100622]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:31 smithi184 ceph-mon[102844]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:15:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:33 smithi196 ceph-mon[76349]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:33 smithi184 ceph-mon[100622]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:33 smithi184 ceph-mon[102844]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:35.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:35 smithi184 ceph-mon[100622]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:35.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:35 smithi184 ceph-mon[102844]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:35 smithi196 ceph-mon[76349]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:36.884 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:36 smithi184 ceph-mon[100622]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:36.884 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:36 smithi184 ceph-mon[102844]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:36 smithi196 ceph-mon[76349]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:39 smithi196 ceph-mon[76349]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[100622]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[102844]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:40.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:40.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:40.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:41 smithi196 ceph-mon[76349]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:41 smithi184 ceph-mon[100622]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:41 smithi184 ceph-mon[102844]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:15:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:43 smithi196 ceph-mon[76349]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:43 smithi184 ceph-mon[100622]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:43 smithi184 ceph-mon[102844]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:45 smithi196 ceph-mon[76349]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:45 smithi184 ceph-mon[100622]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:45 smithi184 ceph-mon[102844]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:47 smithi196 ceph-mon[76349]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:47 smithi184 ceph-mon[100622]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:47 smithi184 ceph-mon[102844]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:49 smithi196 ceph-mon[76349]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:49 smithi184 ceph-mon[100622]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:49 smithi184 ceph-mon[102844]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:50.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:50.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:15:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:15:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:15:50.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:15:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:51 smithi196 ceph-mon[76349]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:51 smithi184 ceph-mon[100622]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:51 smithi184 ceph-mon[102844]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:15:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:15:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:53 smithi196 ceph-mon[76349]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:53 smithi184 ceph-mon[100622]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:53 smithi184 ceph-mon[102844]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:55 smithi196 ceph-mon[76349]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:55.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:55 smithi184 ceph-mon[100622]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:55.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:55 smithi184 ceph-mon[102844]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:55.785 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:15:56.318 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:15:56.318 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T22:15:56.318 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T22:15:56.318 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 19s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:15:56.318 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 19s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 19s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 19s ago 2h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 2h 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 19s ago 2h 195M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 2h 18.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 19s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 19s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 19s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 19s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:15:56.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 17s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 17s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 73.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 19s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 17s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 19s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:15:56.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 17s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:15:56.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:15:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:15:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:15:56.642 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:15:56.912 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:15:56.912 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:15:56.912 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:15:56.912 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:15:56.913 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:15:57.236 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:15:57.236 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:57.236 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='client.63722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='client.63734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1662991264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3317550184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='client.63722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='client.63734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1662991264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3317550184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:57.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='client.63722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='client.63734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1662991264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3317550184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:58 smithi196 ceph-mon[76349]: from='client.63746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:58 smithi184 ceph-mon[100622]: from='client.63746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:58 smithi184 ceph-mon[102844]: from='client.63746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[100622]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[102844]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:59 smithi196 ceph-mon[76349]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:15:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:00.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:00.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:00.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:01 smithi196 ceph-mon[76349]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:01 smithi184 ceph-mon[100622]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:01 smithi184 ceph-mon[102844]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:03 smithi196 ceph-mon[76349]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:03 smithi184 ceph-mon[100622]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:03 smithi184 ceph-mon[102844]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:05 smithi196 ceph-mon[76349]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:05 smithi184 ceph-mon[100622]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:05 smithi184 ceph-mon[102844]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:07 smithi196 ceph-mon[76349]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:07 smithi184 ceph-mon[100622]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:07 smithi184 ceph-mon[102844]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:09 smithi196 ceph-mon[76349]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:09 smithi184 ceph-mon[102844]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:09 smithi184 ceph-mon[100622]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:10.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:10.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:10.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:11 smithi196 ceph-mon[76349]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:11 smithi184 ceph-mon[100622]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:11 smithi184 ceph-mon[102844]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:16:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:13 smithi196 ceph-mon[76349]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:13 smithi184 ceph-mon[100622]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:13 smithi184 ceph-mon[102844]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:14.686 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:15 smithi184 ceph-mon[100622]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:15 smithi184 ceph-mon[102844]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:15 smithi196 ceph-mon[76349]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[100622]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[102844]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:17 smithi196 ceph-mon[76349]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[100622]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[102844]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:19 smithi196 ceph-mon[76349]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:20.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:20.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:20.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:21.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:21 smithi196 ceph-mon[76349]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:21 smithi184 ceph-mon[100622]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:21 smithi184 ceph-mon[102844]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:23 smithi196 ceph-mon[76349]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:23 smithi184 ceph-mon[100622]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:23 smithi184 ceph-mon[102844]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:25 smithi196 ceph-mon[76349]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:25 smithi184 ceph-mon[100622]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:25 smithi184 ceph-mon[102844]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:27.536 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:16:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:27 smithi196 ceph-mon[76349]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:27 smithi184 ceph-mon[100622]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:27 smithi184 ceph-mon[102844]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:28.063 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 2h - - 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 2h - - 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 2h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 2h 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:16:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 10s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 10s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 9s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 9s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 2h 73.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 9s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:16:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 9s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:16:28.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:16:28.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:16:28.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:16:28.380 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:16:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:28 smithi196 ceph-mon[76349]: from='client.63758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:16:28.649 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:16:28.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:16:28.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:28 smithi184 ceph-mon[100622]: from='client.63758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:28.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:28 smithi184 ceph-mon[102844]: from='client.63758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:28.961 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:16:28.961 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:28.962 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: from='client.70171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2875482981' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3234342909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: from='client.70171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2875482981' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3234342909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: from='client.70171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2875482981' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3234342909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:30 smithi196 ceph-mon[76349]: from='client.53817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:30.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:30 smithi184 ceph-mon[100622]: from='client.53817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:30.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:30 smithi184 ceph-mon[102844]: from='client.53817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:30.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:30.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:30.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:31 smithi196 ceph-mon[76349]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:31 smithi184 ceph-mon[100622]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:31 smithi184 ceph-mon[102844]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:16:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:33 smithi196 ceph-mon[76349]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:33.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:33 smithi184 ceph-mon[100622]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:33.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:33 smithi184 ceph-mon[102844]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:35 smithi196 ceph-mon[76349]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:35 smithi184 ceph-mon[100622]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:35 smithi184 ceph-mon[102844]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:37.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:37 smithi184 ceph-mon[100622]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:37 smithi196 ceph-mon[76349]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:37.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:37 smithi184 ceph-mon[102844]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:38.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[100622]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[102844]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:39 smithi196 ceph-mon[76349]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:40.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:40.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:40.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:41 smithi184 ceph-mon[100622]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:41 smithi184 ceph-mon[102844]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:41.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:41 smithi196 ceph-mon[76349]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:16:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:43 smithi184 ceph-mon[100622]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:43 smithi184 ceph-mon[102844]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:43 smithi196 ceph-mon[76349]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:45 smithi184 ceph-mon[100622]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:45 smithi184 ceph-mon[102844]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:45 smithi196 ceph-mon[76349]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:47 smithi184 ceph-mon[100622]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:47 smithi184 ceph-mon[102844]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:47 smithi196 ceph-mon[76349]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:49 smithi184 ceph-mon[100622]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:49 smithi184 ceph-mon[102844]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:49 smithi196 ceph-mon[76349]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:50.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:50.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:16:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:16:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:16:50.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:16:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:51 smithi184 ceph-mon[100622]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:51 smithi184 ceph-mon[102844]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:51 smithi196 ceph-mon[76349]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:16:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:16:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:53 smithi184 ceph-mon[100622]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:53 smithi184 ceph-mon[102844]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:53 smithi196 ceph-mon[76349]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:55 smithi184 ceph-mon[100622]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:55 smithi184 ceph-mon[102844]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:55 smithi196 ceph-mon[76349]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:57 smithi184 ceph-mon[100622]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:57 smithi184 ceph-mon[102844]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:57 smithi196 ceph-mon[76349]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:16:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:16:59.816 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:16:59.816 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T22:16:59.816 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 2h - - 2024-09-20T22:16:59.816 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 0s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 2s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 2s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 2s ago 2h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 2s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 2s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 2s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 2s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 2s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:16:59.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 0s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 0s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 0s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 2h 76.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 2s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 0s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 2s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:16:59.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 0s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[100622]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[102844]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:59 smithi196 ceph-mon[76349]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:17:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:17:00.143 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:17:00.416 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:17:00.416 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:17:00.417 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[100622]: from='client.63794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[100622]: from='client.63800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1968080341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[102844]: from='client.63794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[102844]: from='client.63800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:00.723 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1968080341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:17:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:00.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:00.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:00.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:00 smithi196 ceph-mon[76349]: from='client.63794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:00 smithi196 ceph-mon[76349]: from='client.63800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1968080341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:01.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:01 smithi196 ceph-mon[76349]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:01.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:01 smithi196 ceph-mon[76349]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:01.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:01 smithi196 ceph-mon[76349]: from='client.53853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:01.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4279430934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[100622]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[100622]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[100622]: from='client.53853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4279430934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[102844]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[102844]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[102844]: from='client.53853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4279430934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:17:02.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:02 smithi196 ceph-mon[76349]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:02.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:02 smithi184 ceph-mon[100622]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:02 smithi184 ceph-mon[102844]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:05 smithi196 ceph-mon[76349]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:05 smithi184 ceph-mon[100622]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:05 smithi184 ceph-mon[102844]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:07 smithi196 ceph-mon[76349]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:07 smithi184 ceph-mon[100622]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:07 smithi184 ceph-mon[102844]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:09 smithi196 ceph-mon[76349]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:09 smithi184 ceph-mon[100622]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:09 smithi184 ceph-mon[102844]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:10.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:10.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:10.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:11 smithi196 ceph-mon[76349]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:11 smithi184 ceph-mon[100622]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:11 smithi184 ceph-mon[102844]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:17:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:13 smithi196 ceph-mon[76349]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:13 smithi184 ceph-mon[100622]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:13 smithi184 ceph-mon[102844]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:15 smithi196 ceph-mon[76349]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:15 smithi184 ceph-mon[100622]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:15 smithi184 ceph-mon[102844]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:17 smithi196 ceph-mon[76349]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:17.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:17 smithi184 ceph-mon[100622]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:17.771 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:17 smithi184 ceph-mon[102844]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[100622]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:18 smithi184 ceph-mon[102844]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:18 smithi196 ceph-mon[76349]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:20.725 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:20.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:20.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:20.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:21.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:21 smithi184 ceph-mon[100622]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:21.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:21 smithi184 ceph-mon[102844]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:21 smithi196 ceph-mon[76349]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:17:23.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:23 smithi196 ceph-mon[76349]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:23.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:23 smithi184 ceph-mon[102844]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:23 smithi184 ceph-mon[100622]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:25 smithi184 ceph-mon[100622]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:25 smithi184 ceph-mon[102844]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:25 smithi196 ceph-mon[76349]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:27 smithi184 ceph-mon[100622]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:27 smithi184 ceph-mon[102844]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:27 smithi196 ceph-mon[76349]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:29 smithi184 ceph-mon[100622]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:29 smithi184 ceph-mon[102844]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:29 smithi196 ceph-mon[76349]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:31.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:30.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:31.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:30.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:31.024 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:30.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:31.025 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:17:31.550 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:17:31.550 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 2h - - 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 13s ago 2h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 12s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 13s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 13s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 13s ago 2h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 12s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 13s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 13s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 12s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 13s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 13s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:17:31.551 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 12s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 12s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 12s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 12s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 12s ago 2h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 12s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:17:31.552 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 12s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:17:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:31 smithi184 ceph-mon[100622]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:31 smithi184 ceph-mon[102844]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:17:31.863 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:17:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:31 smithi196 ceph-mon[76349]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:17:32.134 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:17:32.135 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:17:32.135 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:17:32.135 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:17:32.135 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:17:32.135 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:17:32.135 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:17:32.447 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:17:32.447 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:32.447 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:32.447 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:32.447 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[100622]: from='client.63830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[100622]: from='client.53871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[100622]: from='client.63842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/863172508' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[102844]: from='client.63830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[102844]: from='client.53871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[102844]: from='client.63842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/863172508' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:32 smithi196 ceph-mon[76349]: from='client.63830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:32 smithi196 ceph-mon[76349]: from='client.53871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:32 smithi196 ceph-mon[76349]: from='client.63842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/863172508' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[100622]: from='client.63854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[100622]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1889519895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[102844]: from='client.63854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[102844]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1889519895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:33 smithi196 ceph-mon[76349]: from='client.63854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:33 smithi196 ceph-mon[76349]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1889519895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:35 smithi184 ceph-mon[102844]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:35 smithi184 ceph-mon[100622]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:35 smithi196 ceph-mon[76349]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:37 smithi184 ceph-mon[100622]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:37 smithi184 ceph-mon[102844]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:37 smithi196 ceph-mon[76349]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:38 smithi196 ceph-mon[76349]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[100622]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:38 smithi184 ceph-mon[102844]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:40.990 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:40.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:40.990 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:40.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:40.991 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:40.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:41 smithi184 ceph-mon[100622]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:41 smithi184 ceph-mon[102844]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:41 smithi196 ceph-mon[76349]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:42.139 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:17:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:43 smithi196 ceph-mon[76349]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:43 smithi184 ceph-mon[100622]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:43 smithi184 ceph-mon[102844]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:44.843 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:45 smithi184 ceph-mon[100622]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:45 smithi184 ceph-mon[102844]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:45 smithi196 ceph-mon[76349]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:47 smithi184 ceph-mon[100622]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:47 smithi184 ceph-mon[102844]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:47 smithi196 ceph-mon[76349]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:49 smithi184 ceph-mon[100622]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:49 smithi184 ceph-mon[102844]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:49 smithi196 ceph-mon[76349]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:50.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:50.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:17:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:17:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:17:50.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:17:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:51 smithi184 ceph-mon[100622]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:51 smithi184 ceph-mon[102844]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:51 smithi196 ceph-mon[76349]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:17:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:17:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:53 smithi184 ceph-mon[100622]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:53 smithi184 ceph-mon[102844]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:53 smithi196 ceph-mon[76349]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:55 smithi184 ceph-mon[100622]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:55 smithi184 ceph-mon[102844]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:55 smithi196 ceph-mon[76349]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:57 smithi196 ceph-mon[76349]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:58.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:57 smithi184 ceph-mon[100622]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:58.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:57 smithi184 ceph-mon[102844]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[100622]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[102844]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:17:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:59 smithi196 ceph-mon[76349]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:00.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.862 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:00.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:00.862 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:00.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:00.862 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:00.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:01 smithi184 ceph-mon[100622]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:01 smithi184 ceph-mon[102844]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:01 smithi196 ceph-mon[76349]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:02.287 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:18:02.744 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 2h - - 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 3s ago 2h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:18:03.284 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 2h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 3s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:18:03.285 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 3s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 3s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 2h 76.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:18:03.286 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 3s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:18:03.595 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:18:03.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:03.596 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:18:03.596 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:18:03.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:18:03.596 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:18:03.865 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:18:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:18:03.867 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[100622]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2365185823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[102844]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:03.867 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2365185823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:03 smithi196 ceph-mon[76349]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2365185823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:04.178 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:18:04.178 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:04.178 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:04.178 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:04.178 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[102844]: from='client.63866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[102844]: from='client.63872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[102844]: from='client.63878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3149295968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[100622]: from='client.63866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[100622]: from='client.63872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[100622]: from='client.63878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3149295968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:04 smithi196 ceph-mon[76349]: from='client.63866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:04 smithi196 ceph-mon[76349]: from='client.63872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:04 smithi196 ceph-mon[76349]: from='client.63878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3149295968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:05 smithi184 ceph-mon[102844]: from='client.63890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:05 smithi184 ceph-mon[102844]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:05 smithi184 ceph-mon[100622]: from='client.63890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:05 smithi184 ceph-mon[100622]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:05 smithi196 ceph-mon[76349]: from='client.63890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:05 smithi196 ceph-mon[76349]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:07 smithi184 ceph-mon[102844]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:07 smithi184 ceph-mon[100622]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:07 smithi196 ceph-mon[76349]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:08 smithi184 ceph-mon[102844]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:08 smithi184 ceph-mon[100622]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:08 smithi196 ceph-mon[76349]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:10.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:10.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:10.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:11 smithi196 ceph-mon[76349]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:11 smithi184 ceph-mon[100622]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:11 smithi184 ceph-mon[102844]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:18:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:13 smithi196 ceph-mon[76349]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:13 smithi184 ceph-mon[100622]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:13 smithi184 ceph-mon[102844]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:18:13.810972286Z level=info msg="Completed cleanup jobs" duration=11.149971ms 2024-09-20T22:18:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:15 smithi196 ceph-mon[76349]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:15 smithi184 ceph-mon[100622]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:15 smithi184 ceph-mon[102844]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:17 smithi184 ceph-mon[100622]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:17 smithi184 ceph-mon[102844]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:17 smithi196 ceph-mon[76349]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[100622]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:19 smithi184 ceph-mon[102844]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:19 smithi196 ceph-mon[76349]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[100622]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:20.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-mon[102844]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:20.991 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:20.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:20.991 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:20.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:20.991 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:20.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:20 smithi196 ceph-mon[76349]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:18:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:23 smithi196 ceph-mon[76349]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:23.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:23 smithi184 ceph-mon[100622]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:23.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:23.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:23 smithi184 ceph-mon[102844]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:23.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:25 smithi196 ceph-mon[76349]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:25 smithi184 ceph-mon[100622]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:25 smithi184 ceph-mon[102844]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:27 smithi196 ceph-mon[76349]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:27 smithi184 ceph-mon[100622]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:27 smithi184 ceph-mon[102844]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:29 smithi184 ceph-mon[100622]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:29 smithi184 ceph-mon[102844]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:29 smithi196 ceph-mon[76349]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:30.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:30.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:30.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:31 smithi184 ceph-mon[100622]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:31 smithi184 ceph-mon[102844]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:31 smithi196 ceph-mon[76349]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:18:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:33 smithi184 ceph-mon[100622]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:33 smithi184 ceph-mon[102844]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:33 smithi196 ceph-mon[76349]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:34.466 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 16s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 15s ago 2h 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 16s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 16s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 16s ago 2h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 15s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:18:34.993 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 16s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 16s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 15s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 16s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 16s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 16s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 16s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 15s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 15s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 15s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 15s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 15s ago 2h 77.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 15s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:18:34.994 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:18:34.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 15s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:18:35.300 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:18:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:18:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:18:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:18:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:18:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:18:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:18:35.302 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:18:35.568 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:18:35.569 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:18:35.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[100622]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:35.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[100622]: from='client.63902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:35.569 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3951371750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:35.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[102844]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:35.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[102844]: from='client.63902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:35.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3951371750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:35.876 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:18:35.877 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:35.877 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:35.877 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:35.877 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:35 smithi196 ceph-mon[76349]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:35 smithi196 ceph-mon[76349]: from='client.63902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3951371750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[100622]: from='client.53943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[100622]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[100622]: from='client.63926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1611646604' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[102844]: from='client.53943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[102844]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[102844]: from='client.63926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1611646604' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:36 smithi196 ceph-mon[76349]: from='client.53943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:36 smithi196 ceph-mon[76349]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:36 smithi196 ceph-mon[76349]: from='client.63926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1611646604' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:37 smithi184 ceph-mon[100622]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:37 smithi184 ceph-mon[102844]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:37 smithi196 ceph-mon[76349]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[100622]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:39 smithi184 ceph-mon[102844]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:39 smithi196 ceph-mon[76349]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[100622]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-mon[102844]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:40.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:40.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:40.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:40 smithi196 ceph-mon[76349]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:42.150 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:18:43.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:43 smithi184 ceph-mon[100622]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:43.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:43.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:43 smithi184 ceph-mon[102844]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:43.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:43 smithi196 ceph-mon[76349]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:44.461 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:44.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:44.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:45.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:45 smithi196 ceph-mon[76349]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:45 smithi184 ceph-mon[100622]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:45 smithi184 ceph-mon[102844]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:47 smithi184 ceph-mon[100622]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:47 smithi184 ceph-mon[102844]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:47 smithi196 ceph-mon[76349]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:49 smithi184 ceph-mon[100622]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:49 smithi184 ceph-mon[102844]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:49 smithi196 ceph-mon[76349]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:50.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:50.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:18:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:18:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:18:50.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:18:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:51 smithi184 ceph-mon[100622]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:51 smithi184 ceph-mon[102844]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:51 smithi196 ceph-mon[76349]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:18:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:18:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:53 smithi184 ceph-mon[100622]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:53 smithi184 ceph-mon[102844]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:53 smithi196 ceph-mon[76349]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:55 smithi184 ceph-mon[100622]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:55 smithi184 ceph-mon[102844]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:55 smithi196 ceph-mon[76349]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:57 smithi184 ceph-mon[100622]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:57 smithi184 ceph-mon[102844]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:57 smithi196 ceph-mon[76349]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[100622]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[102844]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:59 smithi196 ceph-mon[76349]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:18:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:01.020 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:00.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:01.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:00.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:01.021 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:00.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[100622]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:01 smithi184 ceph-mon[102844]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:01 smithi196 ceph-mon[76349]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:19:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:03 smithi184 ceph-mon[100622]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:03 smithi184 ceph-mon[102844]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:03 smithi196 ceph-mon[76349]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:05.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:05 smithi196 ceph-mon[76349]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:05 smithi184 ceph-mon[100622]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:05 smithi184 ceph-mon[102844]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:06.170 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:19:06.700 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 2h - - 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 6s ago 2h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 2h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 16.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 6s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:19:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 6s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 6s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 6s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 2h 77.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:19:06.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:07.012 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:19:07.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:19:07.281 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:19:07.282 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:19:07.282 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[100622]: from='client.63938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[100622]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[100622]: from='client.53979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3374625404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[102844]: from='client.63938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[102844]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[102844]: from='client.53979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3374625404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:07.594 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:19:07.594 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:07.594 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:07.594 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:07.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:07 smithi196 ceph-mon[76349]: from='client.63938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:07 smithi196 ceph-mon[76349]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:07.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:07 smithi196 ceph-mon[76349]: from='client.53979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:07.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3374625404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[100622]: from='client.63950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[100622]: from='client.63962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2276870541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[102844]: from='client.63950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[102844]: from='client.63962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2276870541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:08 smithi196 ceph-mon[76349]: from='client.63950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:08 smithi196 ceph-mon[76349]: from='client.63962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2276870541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:09 smithi184 ceph-mon[102844]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:09 smithi184 ceph-mon[100622]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:09 smithi196 ceph-mon[76349]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:10.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:10.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:10.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:11 smithi184 ceph-mon[100622]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:11 smithi184 ceph-mon[102844]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:11 smithi196 ceph-mon[76349]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:19:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:13 smithi196 ceph-mon[76349]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:13 smithi184 ceph-mon[100622]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:13 smithi184 ceph-mon[102844]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:15 smithi196 ceph-mon[76349]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:15 smithi184 ceph-mon[100622]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:15 smithi184 ceph-mon[102844]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:17.663 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:17 smithi184 ceph-mon[100622]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:17.663 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:17 smithi184 ceph-mon[102844]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:17 smithi196 ceph-mon[76349]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[100622]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[102844]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:19 smithi196 ceph-mon[76349]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:20.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:20.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:20.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[100622]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:21 smithi184 ceph-mon[102844]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:21 smithi196 ceph-mon[76349]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:19:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:23 smithi184 ceph-mon[100622]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:23 smithi184 ceph-mon[102844]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:23 smithi196 ceph-mon[76349]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:25 smithi184 ceph-mon[100622]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:25 smithi184 ceph-mon[102844]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:25.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:25 smithi196 ceph-mon[76349]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:27.520 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:27 smithi196 ceph-mon[76349]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:27 smithi184 ceph-mon[100622]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:27 smithi184 ceph-mon[102844]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:29 smithi184 ceph-mon[102844]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:29 smithi184 ceph-mon[100622]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:29 smithi196 ceph-mon[76349]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:30.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:30.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:30.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:31 smithi184 ceph-mon[100622]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:31 smithi184 ceph-mon[102844]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:31 smithi196 ceph-mon[76349]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:19:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:33 smithi184 ceph-mon[100622]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:33 smithi184 ceph-mon[102844]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:33 smithi196 ceph-mon[76349]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:35.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:35 smithi184 ceph-mon[100622]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:35.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:35 smithi184 ceph-mon[102844]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:35 smithi196 ceph-mon[76349]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:37 smithi184 ceph-mon[100622]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:37 smithi184 ceph-mon[102844]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:37 smithi196 ceph-mon[76349]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.902 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 19s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 19s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:19:38.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 19s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 19s ago 2h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 19s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 19s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 19s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 19s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 19s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:19:38.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 2h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 19s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 19s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:19:38.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 2h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:19:38.779 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:19:38.780 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:19:38.781 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:19:38.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:19:38.781 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:19:39.046 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:19:39.047 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:19:39.356 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:19:39.356 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:39.356 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:39.356 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:39.356 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='client.54009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='client.54015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='client.63986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/717026089' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2672845750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='client.54009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='client.54015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='client.63986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/717026089' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2672845750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='client.54009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='client.54015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='client.63986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/717026089' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2672845750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[102844]: from='client.63998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[100622]: from='client.63998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:40.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:40.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:40.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:40 smithi196 ceph-mon[76349]: from='client.63998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:19:42.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:41 smithi184 ceph-mon[100622]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:41 smithi184 ceph-mon[102844]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:41 smithi196 ceph-mon[76349]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:19:43.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:43 smithi196 ceph-mon[76349]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:43.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:43.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:43 smithi184 ceph-mon[100622]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:43.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:43.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:43 smithi184 ceph-mon[102844]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:43.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:45.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:45 smithi196 ceph-mon[76349]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:46.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:45 smithi184 ceph-mon[100622]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:46.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:45 smithi184 ceph-mon[102844]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:47 smithi184 ceph-mon[100622]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:47 smithi184 ceph-mon[102844]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:47 smithi196 ceph-mon[76349]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:49.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:49 smithi196 ceph-mon[76349]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:49 smithi184 ceph-mon[100622]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:49 smithi184 ceph-mon[102844]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:50.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:50.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:19:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:19:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:19:50.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:19:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:51 smithi184 ceph-mon[100622]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:51 smithi184 ceph-mon[102844]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:51 smithi196 ceph-mon[76349]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:19:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:19:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:53 smithi184 ceph-mon[100622]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:53 smithi184 ceph-mon[102844]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:53 smithi196 ceph-mon[76349]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:55 smithi184 ceph-mon[100622]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:55 smithi184 ceph-mon[102844]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:55 smithi196 ceph-mon[76349]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:57 smithi184 ceph-mon[100622]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:57 smithi184 ceph-mon[102844]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:57 smithi196 ceph-mon[76349]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:58 smithi184 ceph-mon[100622]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:58 smithi184 ceph-mon[102844]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:58 smithi196 ceph-mon[76349]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:00.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:00.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:00.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:00.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[100622]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[102844]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:00 smithi196 ceph-mon[76349]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:20:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:03 smithi184 ceph-mon[102844]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:03 smithi184 ceph-mon[100622]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:03 smithi196 ceph-mon[76349]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:05.665 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:05 smithi184 ceph-mon[100622]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:05.666 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:05 smithi184 ceph-mon[102844]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:05 smithi196 ceph-mon[76349]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:07 smithi196 ceph-mon[76349]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:07 smithi184 ceph-mon[100622]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:07 smithi184 ceph-mon[102844]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:09.654 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:20:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:09 smithi184 ceph-mon[100622]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:09 smithi184 ceph-mon[102844]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:09 smithi196 ceph-mon[76349]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 2h - - 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 11s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:20:10.181 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 2h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 9s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 9s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 2h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:20:10.182 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 11s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:20:10.183 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 9s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:20:10.183 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 11s ago 2h 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:20:10.183 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 9s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:20:10.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:10 smithi184 ceph-mon[100622]: from='client.54045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:10.495 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:20:10.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:20:10.497 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:20:10.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:10 smithi184 ceph-mon[102844]: from='client.54045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:20:10.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:20:10.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:20:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:10.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:10.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:10.768 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:10.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:10 smithi196 ceph-mon[76349]: from='client.54045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.076 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:11.076 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:11.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:11.077 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:11.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[100622]: from='client.64016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[100622]: from='client.64022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[100622]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2920506772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1264790649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[102844]: from='client.64016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[102844]: from='client.64022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[102844]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2920506772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1264790649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:11 smithi196 ceph-mon[76349]: from='client.64016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:11 smithi196 ceph-mon[76349]: from='client.64022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:11 smithi196 ceph-mon[76349]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2920506772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1264790649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:20:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:12 smithi184 ceph-mon[100622]: from='client.64034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:12 smithi184 ceph-mon[102844]: from='client.64034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:12 smithi196 ceph-mon[76349]: from='client.64034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:13.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:13 smithi196 ceph-mon[76349]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:13.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:13 smithi184 ceph-mon[100622]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:13 smithi184 ceph-mon[102844]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:15 smithi184 ceph-mon[100622]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:15 smithi184 ceph-mon[102844]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:15 smithi196 ceph-mon[76349]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:17 smithi184 ceph-mon[100622]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:17 smithi184 ceph-mon[102844]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:17 smithi196 ceph-mon[76349]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[100622]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[102844]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:19 smithi196 ceph-mon[76349]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:20.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:20.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:20.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[100622]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[102844]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:21 smithi196 ceph-mon[76349]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:20:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:23 smithi184 ceph-mon[100622]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:23 smithi184 ceph-mon[102844]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:23 smithi196 ceph-mon[76349]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:25.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:25 smithi184 ceph-mon[100622]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:25.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:25 smithi184 ceph-mon[102844]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:25 smithi196 ceph-mon[76349]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:27.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:27 smithi184 ceph-mon[100622]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:27.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:27 smithi184 ceph-mon[102844]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:27 smithi196 ceph-mon[76349]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:29 smithi196 ceph-mon[76349]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.684 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:29 smithi184 ceph-mon[100622]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:29 smithi184 ceph-mon[102844]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:30.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:30.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:30.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:31 smithi196 ceph-mon[76349]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:31 smithi184 ceph-mon[100622]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:31 smithi184 ceph-mon[102844]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:20:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:33 smithi184 ceph-mon[100622]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:33 smithi184 ceph-mon[102844]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:33 smithi196 ceph-mon[76349]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:35 smithi196 ceph-mon[76349]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:35 smithi184 ceph-mon[100622]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:35 smithi184 ceph-mon[102844]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:37 smithi196 ceph-mon[76349]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:37 smithi184 ceph-mon[100622]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:37 smithi184 ceph-mon[102844]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:39 smithi196 ceph-mon[76349]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[100622]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[102844]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:40.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:40.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:40.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:41.371 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:20:41.903 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:20:41.903 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T22:20:41.903 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 2s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 2s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 2s ago 2h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 2s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 2s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 2s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 2s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 2s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:20:41.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 2h 77.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 2s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 2s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:20:41.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:20:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[100622]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[102844]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:41 smithi196 ceph-mon[76349]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:42.211 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:20:42.211 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:20:42.211 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:20:42.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:20:42.212 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:20:42.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:20:42.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:20:42.213 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:20:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:20:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:20:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:20:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:20:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:20:42.482 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:20:42.795 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:42.795 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:42.796 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[100622]: from='client.64046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[100622]: from='client.54087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/43523991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[102844]: from='client.64046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[102844]: from='client.54087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:42.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/43523991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:43.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:42 smithi196 ceph-mon[76349]: from='client.64046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:42 smithi196 ceph-mon[76349]: from='client.54087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/43523991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[102844]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[102844]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[102844]: from='client.64070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/402555873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[100622]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[100622]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[100622]: from='client.64070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/402555873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:44.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:43 smithi196 ceph-mon[76349]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:44.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:43 smithi196 ceph-mon[76349]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:44.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:43 smithi196 ceph-mon[76349]: from='client.64070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:44.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:44.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/402555873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:46.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:45 smithi184 ceph-mon[100622]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:46.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:45 smithi184 ceph-mon[102844]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:45 smithi196 ceph-mon[76349]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:46.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:46 smithi184 ceph-mon[100622]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:46.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:46 smithi184 ceph-mon[102844]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:46 smithi196 ceph-mon[76349]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:49 smithi196 ceph-mon[76349]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:49 smithi184 ceph-mon[100622]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:49 smithi184 ceph-mon[102844]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:50.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:50.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:20:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:20:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:20:50.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:20:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:51 smithi184 ceph-mon[102844]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:51 smithi184 ceph-mon[100622]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:51 smithi196 ceph-mon[76349]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:20:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:20:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:53 smithi184 ceph-mon[102844]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:53 smithi184 ceph-mon[100622]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:53 smithi196 ceph-mon[76349]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:54.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:55 smithi196 ceph-mon[76349]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:55 smithi184 ceph-mon[102844]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:55 smithi184 ceph-mon[100622]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:57 smithi184 ceph-mon[102844]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:57 smithi184 ceph-mon[100622]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:57 smithi196 ceph-mon[76349]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[100622]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[102844]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:59 smithi196 ceph-mon[76349]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:00.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:00.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:00.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[102844]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[100622]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:01 smithi196 ceph-mon[76349]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:21:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:03 smithi184 ceph-mon[100622]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:03 smithi184 ceph-mon[102844]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:03 smithi196 ceph-mon[76349]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:05 smithi184 ceph-mon[100622]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:05 smithi184 ceph-mon[102844]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:05 smithi196 ceph-mon[76349]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:07.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:07 smithi184 ceph-mon[100622]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:07.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:07 smithi184 ceph-mon[102844]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:07 smithi196 ceph-mon[76349]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:09 smithi184 ceph-mon[100622]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:09 smithi184 ceph-mon[102844]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:09.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:09 smithi196 ceph-mon[76349]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:10.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:10.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:10.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:11 smithi196 ceph-mon[76349]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:11 smithi184 ceph-mon[100622]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:11 smithi184 ceph-mon[102844]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:21:13.097 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:21:13.632 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 12s ago 2h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 2h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 12s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:21:13.633 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 12s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 12s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 12s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 12s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 12s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 12s ago 2h 77.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 12s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:21:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:21:13.635 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 12s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:21:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:13 smithi184 ceph-mon[100622]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:13 smithi184 ceph-mon[102844]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:13 smithi196 ceph-mon[76349]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:21:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:21:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:21:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:21:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:21:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:21:14.233 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:21:14.547 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:21:14.547 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:14.548 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='client.64082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='client.64088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='client.64094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4002707082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='client.64082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='client.64088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='client.64094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4002707082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='client.64082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='client.64088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='client.64094 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4002707082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[100622]: from='client.64106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[100622]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4089442806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[102844]: from='client.64106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[102844]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4089442806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:15 smithi196 ceph-mon[76349]: from='client.64106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:15 smithi196 ceph-mon[76349]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4089442806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:17 smithi184 ceph-mon[100622]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:17 smithi184 ceph-mon[102844]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:17 smithi196 ceph-mon[76349]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[100622]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[102844]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:19 smithi196 ceph-mon[76349]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:20.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:21.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:20.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:21.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:20.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[100622]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[102844]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:21 smithi196 ceph-mon[76349]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:21:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:23 smithi184 ceph-mon[100622]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:23 smithi184 ceph-mon[102844]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:23 smithi196 ceph-mon[76349]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:25 smithi184 ceph-mon[100622]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:25 smithi184 ceph-mon[102844]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:25 smithi196 ceph-mon[76349]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:27 smithi184 ceph-mon[100622]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:27 smithi184 ceph-mon[102844]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:27 smithi196 ceph-mon[76349]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:29.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:29 smithi196 ceph-mon[76349]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:29.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:29 smithi184 ceph-mon[100622]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:29.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:29 smithi184 ceph-mon[102844]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:29.792 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:30.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:30.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:30.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:31.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:31 smithi196 ceph-mon[76349]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:31 smithi184 ceph-mon[100622]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:31 smithi184 ceph-mon[102844]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:21:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:33 smithi184 ceph-mon[100622]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:33 smithi184 ceph-mon[102844]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:33 smithi196 ceph-mon[76349]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:35 smithi184 ceph-mon[100622]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:35 smithi184 ceph-mon[102844]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:35 smithi196 ceph-mon[76349]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:37 smithi184 ceph-mon[100622]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:37 smithi184 ceph-mon[102844]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:37 smithi196 ceph-mon[76349]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[100622]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[102844]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:39 smithi196 ceph-mon[76349]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:40.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:40.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:40.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[102844]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[100622]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:41 smithi196 ceph-mon[76349]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:21:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:43 smithi184 ceph-mon[100622]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:43 smithi184 ceph-mon[102844]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:43 smithi196 ceph-mon[76349]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:44.842 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:44.842 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:44.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:44.843 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:44.843 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:21:44.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:44.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:21:45.373 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 2h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 4s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 4s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 4s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:21:45.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 4s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 4s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 4s ago 2h 77.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 4s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:21:45.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 4s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:21:45.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:45 smithi184 ceph-mon[100622]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:45.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:45 smithi184 ceph-mon[102844]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:21:45.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:21:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:21:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:45 smithi196 ceph-mon[76349]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:21:45.954 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:21:45.955 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:21:45.955 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:21:46.265 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:21:46.265 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:46.265 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:46.265 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:46.265 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:46 smithi196 ceph-mon[76349]: from='client.64118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:46 smithi196 ceph-mon[76349]: from='client.64124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:46 smithi196 ceph-mon[76349]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/291997309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3154432466' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[100622]: from='client.64118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[100622]: from='client.64124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[100622]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/291997309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3154432466' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[102844]: from='client.64118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[102844]: from='client.64124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[102844]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/291997309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3154432466' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:47 smithi196 ceph-mon[76349]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:47 smithi196 ceph-mon[76349]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:47.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:47 smithi184 ceph-mon[100622]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:47 smithi184 ceph-mon[100622]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:47.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:47 smithi184 ceph-mon[102844]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:47.922 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:47 smithi184 ceph-mon[102844]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:49.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:49 smithi196 ceph-mon[76349]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:50.042 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:49 smithi184 ceph-mon[100622]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:50.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:49 smithi184 ceph-mon[102844]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:50.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:50.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:21:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:21:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:21:50.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:21:51.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:51 smithi196 ceph-mon[76349]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:51 smithi184 ceph-mon[100622]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:51 smithi184 ceph-mon[102844]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:21:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:21:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:53 smithi196 ceph-mon[76349]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:53 smithi184 ceph-mon[100622]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:53 smithi184 ceph-mon[102844]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:55 smithi196 ceph-mon[76349]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:55 smithi184 ceph-mon[100622]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:55 smithi184 ceph-mon[102844]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:57 smithi196 ceph-mon[76349]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:57 smithi184 ceph-mon[100622]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:57 smithi184 ceph-mon[102844]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:59 smithi196 ceph-mon[76349]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[100622]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[102844]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:00.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:00.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:00.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:01 smithi196 ceph-mon[76349]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[102844]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[100622]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:22:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:03 smithi196 ceph-mon[76349]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:03 smithi184 ceph-mon[100622]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:03 smithi184 ceph-mon[102844]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:05 smithi196 ceph-mon[76349]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:05 smithi184 ceph-mon[100622]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:05 smithi184 ceph-mon[102844]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:07 smithi196 ceph-mon[76349]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:07 smithi184 ceph-mon[100622]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:07 smithi184 ceph-mon[102844]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:09 smithi184 ceph-mon[100622]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:09 smithi184 ceph-mon[102844]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:09 smithi196 ceph-mon[76349]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:10.922 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:10.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:10.923 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:10.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:10.923 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:10.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:11 smithi196 ceph-mon[76349]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:11 smithi184 ceph-mon[100622]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:11 smithi184 ceph-mon[102844]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:22:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:13 smithi196 ceph-mon[76349]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:13 smithi184 ceph-mon[100622]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:13 smithi184 ceph-mon[102844]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:14.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:14.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:15 smithi196 ceph-mon[76349]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:15 smithi184 ceph-mon[100622]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:15 smithi184 ceph-mon[102844]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:16.563 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 15s ago 2h 91.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:22:17.089 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 2h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 15s ago 2h 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 15s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 15s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 15s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 15s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 15s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 15s ago 2h 77.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:22:17.090 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 15s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:22:17.091 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:22:17.091 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 15s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:22:17.398 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:22:17.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:22:17.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:22:17.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:22:17.671 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:22:17.671 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:22:17.672 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[100622]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[100622]: from='client.54189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1041385058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[102844]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[102844]: from='client.54189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:17.673 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1041385058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:17 smithi196 ceph-mon[76349]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:17 smithi196 ceph-mon[76349]: from='client.54189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1041385058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:17.982 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:22:17.982 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:17.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:17.982 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:17.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:18 smithi196 ceph-mon[76349]: from='client.64160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:18 smithi196 ceph-mon[76349]: from='client.64166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:18 smithi196 ceph-mon[76349]: from='client.64178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2637816643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[100622]: from='client.64160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[100622]: from='client.64166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[100622]: from='client.64178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2637816643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[102844]: from='client.64160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[102844]: from='client.64166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[102844]: from='client.64178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2637816643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:19 smithi196 ceph-mon[76349]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:19 smithi184 ceph-mon[100622]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:19 smithi184 ceph-mon[102844]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:20.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:20.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:20.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[100622]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[102844]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:21 smithi196 ceph-mon[76349]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:22:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:23 smithi184 ceph-mon[100622]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:23 smithi184 ceph-mon[102844]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:23 smithi196 ceph-mon[76349]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:25.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:25 smithi184 ceph-mon[100622]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:25 smithi184 ceph-mon[102844]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:25 smithi196 ceph-mon[76349]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:26 smithi184 ceph-mon[100622]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:26 smithi184 ceph-mon[102844]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:26 smithi196 ceph-mon[76349]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:29 smithi184 ceph-mon[100622]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:29.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:29 smithi184 ceph-mon[102844]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:29 smithi196 ceph-mon[76349]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:30.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:30.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:30.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:31 smithi184 ceph-mon[102844]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:31 smithi184 ceph-mon[100622]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:31 smithi196 ceph-mon[76349]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:22:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:33 smithi184 ceph-mon[100622]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:33 smithi184 ceph-mon[102844]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:33 smithi196 ceph-mon[76349]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:35 smithi184 ceph-mon[100622]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:35 smithi184 ceph-mon[102844]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:35 smithi196 ceph-mon[76349]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:37 smithi184 ceph-mon[100622]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:37 smithi184 ceph-mon[102844]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:37 smithi196 ceph-mon[76349]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:39 smithi184 ceph-mon[100622]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:39 smithi184 ceph-mon[102844]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:39 smithi196 ceph-mon[76349]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[100622]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-mon[102844]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:40.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:40.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:40.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:40 smithi196 ceph-mon[76349]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:22:42.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:22:43.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:43 smithi196 ceph-mon[76349]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:43 smithi184 ceph-mon[100622]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:43 smithi184 ceph-mon[102844]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:45 smithi184 ceph-mon[100622]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:45 smithi184 ceph-mon[102844]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:45 smithi196 ceph-mon[76349]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:47 smithi184 ceph-mon[100622]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:47 smithi184 ceph-mon[102844]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:47 smithi196 ceph-mon[76349]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:48.278 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 2h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 7s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:22:48.808 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 7s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 7s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 7s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 7s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 7s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:22:48.809 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:22:48.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 7s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:22:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:22:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:22:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:22:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:22:49.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:22:49.393 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:22:49.393 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:22:49.393 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:22:49.393 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:22:49.394 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:22:49.708 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:22:49.708 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:49.708 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:49.708 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:49.708 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[100622]: from='client.64190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[100622]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[100622]: from='client.64196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/827220625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[102844]: from='client.64190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[102844]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[102844]: from='client.64196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/827220625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:49 smithi196 ceph-mon[76349]: from='client.64190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:49 smithi196 ceph-mon[76349]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:49 smithi196 ceph-mon[76349]: from='client.64196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/827220625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[100622]: from='client.64202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[100622]: from='client.64214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2098909108' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[102844]: from='client.64202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[102844]: from='client.64214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2098909108' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:50 smithi196 ceph-mon[76349]: from='client.64202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:50 smithi196 ceph-mon[76349]: from='client.64214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2098909108' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:50.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:50.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:22:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:22:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:22:50.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:22:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:51 smithi184 ceph-mon[100622]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:51 smithi184 ceph-mon[102844]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:51 smithi196 ceph-mon[76349]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:52.190 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:22:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T22:22:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:53 smithi184 ceph-mon[102844]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:53 smithi184 ceph-mon[100622]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:53 smithi196 ceph-mon[76349]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:54.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:55 smithi184 ceph-mon[102844]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:55 smithi184 ceph-mon[100622]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:55 smithi196 ceph-mon[76349]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:57 smithi184 ceph-mon[100622]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:57 smithi184 ceph-mon[102844]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:57 smithi196 ceph-mon[76349]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:59 smithi184 ceph-mon[100622]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:59 smithi184 ceph-mon[102844]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:59 smithi196 ceph-mon[76349]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[100622]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-mon[102844]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:00.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:00.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:00.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:00 smithi196 ceph-mon[76349]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:23:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:03 smithi196 ceph-mon[76349]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:03 smithi184 ceph-mon[100622]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:03 smithi184 ceph-mon[102844]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:05 smithi196 ceph-mon[76349]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:05 smithi184 ceph-mon[100622]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:05 smithi184 ceph-mon[102844]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:07 smithi196 ceph-mon[76349]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:07 smithi184 ceph-mon[100622]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:07 smithi184 ceph-mon[102844]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:09 smithi196 ceph-mon[76349]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:09 smithi184 ceph-mon[100622]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:09 smithi184 ceph-mon[102844]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:10.887 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:10.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:10.888 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:10.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:10.888 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:10.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:11 smithi196 ceph-mon[76349]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:11 smithi184 ceph-mon[100622]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:11 smithi184 ceph-mon[102844]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:23:13.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:13 smithi196 ceph-mon[76349]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:13 smithi184 ceph-mon[100622]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:13 smithi184 ceph-mon[102844]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:15 smithi196 ceph-mon[76349]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:15 smithi184 ceph-mon[102844]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:15 smithi184 ceph-mon[100622]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:17 smithi196 ceph-mon[76349]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:17 smithi184 ceph-mon[100622]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:17 smithi184 ceph-mon[102844]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:19 smithi196 ceph-mon[76349]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:20.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:19 smithi184 ceph-mon[100622]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:20.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:19 smithi184 ceph-mon[102844]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:20.010 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 2h - - 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 0s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:23:20.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 0s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 2h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 0s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:23:20.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:23:20.545 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:23:20.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:23:20.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:23:20.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:23:20.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:23:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:20.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:20.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:20.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:20.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:23:20.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:23:20.883 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:23:21.163 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:23:21.164 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:23:21.165 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='client.64226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='client.64232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:21.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='client.54273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2540169851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='client.64226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='client.64232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='client.54273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2540169851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='client.64226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='client.64232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='client.54273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2540169851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:21.471 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:23:21.471 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:21.471 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:21.471 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:21.471 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:23:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[100622]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2741786862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[102844]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2741786862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:22 smithi196 ceph-mon[76349]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2741786862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:23 smithi196 ceph-mon[76349]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:23 smithi184 ceph-mon[100622]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:23 smithi184 ceph-mon[102844]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:25 smithi196 ceph-mon[76349]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:25 smithi184 ceph-mon[100622]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:25 smithi184 ceph-mon[102844]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:27 smithi184 ceph-mon[100622]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:27 smithi184 ceph-mon[102844]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:27 smithi196 ceph-mon[76349]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:29 smithi184 ceph-mon[100622]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:29 smithi184 ceph-mon[102844]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:29 smithi196 ceph-mon[76349]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:31.039 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:30.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:31.039 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:30.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:31.040 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:30.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:31.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:31 smithi184 ceph-mon[100622]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:31.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:31 smithi184 ceph-mon[102844]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:31 smithi196 ceph-mon[76349]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:23:33.661 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:33 smithi184 ceph-mon[100622]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:33.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:33 smithi184 ceph-mon[102844]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:33.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:33 smithi196 ceph-mon[76349]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:34.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:35 smithi196 ceph-mon[76349]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:35 smithi184 ceph-mon[100622]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:35 smithi184 ceph-mon[102844]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:37 smithi184 ceph-mon[100622]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:37 smithi184 ceph-mon[102844]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:37 smithi196 ceph-mon[76349]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:39 smithi184 ceph-mon[100622]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:39 smithi184 ceph-mon[102844]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:39 smithi196 ceph-mon[76349]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:41.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:40.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:41.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:40.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:41.017 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:40.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[102844]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:41.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:41 smithi184 ceph-mon[100622]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:41 smithi196 ceph-mon[76349]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:23:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:42 smithi196 ceph-mon[76349]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[100622]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:23:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:42 smithi184 ceph-mon[102844]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:44 smithi196 ceph-mon[76349]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:44 smithi184 ceph-mon[100622]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:44 smithi184 ceph-mon[102844]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:47 smithi184 ceph-mon[100622]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:47 smithi184 ceph-mon[102844]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:47 smithi196 ceph-mon[76349]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:49 smithi184 ceph-mon[100622]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:49 smithi184 ceph-mon[102844]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:49 smithi196 ceph-mon[76349]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:50.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:50.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:23:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:23:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:23:50.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:23:51.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:51 smithi184 ceph-mon[100622]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:51.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:51 smithi184 ceph-mon[102844]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:51.780 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:23:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:51 smithi196 ceph-mon[76349]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:23:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 12s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 2h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:23:52.316 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 2h 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 12s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 12s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 12s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 12s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:23:52.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 2h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 10s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 12s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 12s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:23:52.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:23:52.637 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:23:52.914 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:23:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:23:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:23:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:23:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:23:52.915 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:23:53.237 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:23:53.237 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:53.237 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:53.237 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:53.237 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: from='client.54297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: from='client.64268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1937970337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1977122546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: from='client.54297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: from='client.64268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1937970337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1977122546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: from='client.54297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: from='client.64268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1937970337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1977122546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:54.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:54.755 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:54 smithi196 ceph-mon[76349]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:54 smithi184 ceph-mon[102844]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:54 smithi184 ceph-mon[100622]: from='client.54321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:55.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:55 smithi196 ceph-mon[76349]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:55 smithi184 ceph-mon[100622]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:55 smithi184 ceph-mon[102844]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:57 smithi196 ceph-mon[76349]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:57 smithi184 ceph-mon[100622]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:57 smithi184 ceph-mon[102844]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:59 smithi184 ceph-mon[100622]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:59 smithi184 ceph-mon[102844]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:59 smithi196 ceph-mon[76349]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:00.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:00.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:00.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[100622]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:01 smithi184 ceph-mon[102844]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:01 smithi196 ceph-mon[76349]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:24:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[100622]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:02 smithi184 ceph-mon[102844]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:02 smithi196 ceph-mon[76349]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:04 smithi184 ceph-mon[100622]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:24:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:04 smithi184 ceph-mon[102844]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:24:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:04 smithi196 ceph-mon[76349]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:24:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:07 smithi184 ceph-mon[100622]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:24:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:07 smithi184 ceph-mon[102844]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:24:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:07 smithi196 ceph-mon[76349]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:24:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:09 smithi184 ceph-mon[100622]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:24:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:09 smithi184 ceph-mon[102844]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:24:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:09 smithi196 ceph-mon[76349]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:24:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:10.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:10.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:10.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:11 smithi184 ceph-mon[100622]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T22:24:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:11 smithi184 ceph-mon[102844]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T22:24:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:11 smithi196 ceph-mon[76349]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T22:24:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:24:13.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:13 smithi184 ceph-mon[100622]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:24:13.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:13 smithi184 ceph-mon[102844]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:24:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:13 smithi196 ceph-mon[76349]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:24:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:14.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:14.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:15 smithi184 ceph-mon[100622]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:24:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:15 smithi184 ceph-mon[102844]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:24:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:15 smithi196 ceph-mon[76349]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:24:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:17 smithi196 ceph-mon[76349]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:24:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:17 smithi184 ceph-mon[100622]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:24:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:17 smithi184 ceph-mon[102844]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:24:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:19 smithi184 ceph-mon[100622]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T22:24:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:19 smithi184 ceph-mon[102844]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T22:24:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:19 smithi196 ceph-mon[76349]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-20T22:24:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:20.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:20.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:20.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[100622]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[102844]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:21 smithi196 ceph-mon[76349]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:24:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:24:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[102844]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T22:24:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:23 smithi184 ceph-mon[100622]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T22:24:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:23.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:23 smithi196 ceph-mon[76349]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T22:24:23.535 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:24:24.070 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:24:24.070 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 2h - - 2024-09-20T22:24:24.070 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T22:24:24.070 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 3s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:24:24.070 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 3s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 3s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 2h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 2h 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 3s ago 2h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 2h 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 2h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 2h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:24:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 2h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 2h 81.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 2h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:24:24.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:24:24.381 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:24:24.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:24:24.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:24:24.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:24:24.383 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:24:24.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:24.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:24 smithi184 ceph-mon[100622]: from='client.64298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:24.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:24.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:24 smithi184 ceph-mon[102844]: from='client.64298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:24 smithi196 ceph-mon[76349]: from='client.64298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:24:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:24:24.653 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:24:24.960 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:24:24.961 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:24.961 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:24.961 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:24.961 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4229308064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: from='client.54357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3247358574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4229308064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: from='client.54357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3247358574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4229308064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: from='client.54357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3247358574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:27 smithi184 ceph-mon[100622]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:24:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:27 smithi184 ceph-mon[102844]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:24:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:27 smithi196 ceph-mon[76349]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:24:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:29 smithi184 ceph-mon[100622]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:24:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:29 smithi184 ceph-mon[102844]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:24:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:29 smithi196 ceph-mon[76349]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:24:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:30.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:30.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:30.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:31 smithi184 ceph-mon[100622]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:24:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:31 smithi184 ceph-mon[102844]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:24:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:31 smithi196 ceph-mon[76349]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-20T22:24:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:24:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:33 smithi184 ceph-mon[100622]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:33 smithi184 ceph-mon[102844]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:33 smithi196 ceph-mon[76349]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:24:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:35.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:35 smithi184 ceph-mon[100622]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:24:35.711 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:35 smithi184 ceph-mon[102844]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:24:35.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:35 smithi196 ceph-mon[76349]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:24:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:37 smithi184 ceph-mon[100622]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:24:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:37 smithi184 ceph-mon[102844]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:24:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:37 smithi196 ceph-mon[76349]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:24:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:39 smithi184 ceph-mon[100622]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:24:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:39 smithi184 ceph-mon[102844]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:24:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:39 smithi196 ceph-mon[76349]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:24:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:40.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:40.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:40.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:41 smithi196 ceph-mon[76349]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:24:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[100622]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:24:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[102844]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T22:24:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:24:43.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:43 smithi196 ceph-mon[76349]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[100622]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:43 smithi184 ceph-mon[102844]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T22:24:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:44.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:44.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:45 smithi184 ceph-mon[100622]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:45 smithi184 ceph-mon[102844]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:45 smithi196 ceph-mon[76349]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:47 smithi184 ceph-mon[100622]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:47 smithi184 ceph-mon[102844]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:47 smithi196 ceph-mon[76349]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:49 smithi184 ceph-mon[100622]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:49 smithi184 ceph-mon[102844]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:49 smithi196 ceph-mon[76349]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:50.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:50.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:24:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:24:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:24:50.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:24:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:51 smithi184 ceph-mon[100622]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:51 smithi184 ceph-mon[102844]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:51 smithi196 ceph-mon[76349]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:24:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:24:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:53 smithi184 ceph-mon[100622]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:53 smithi184 ceph-mon[102844]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:53 smithi196 ceph-mon[76349]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:55.271 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:24:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:55 smithi184 ceph-mon[100622]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:55 smithi184 ceph-mon[102844]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:24:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T22:24:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 2h - - 2024-09-20T22:24:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:24:55.822 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 13s ago 2h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 13s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 15s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 15s ago 2h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 2h 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 13s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 15s ago 2h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 15s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 15s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 15s ago 2h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:24:55.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 13s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 15s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:24:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:24:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:55 smithi196 ceph-mon[76349]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:56.142 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:24:56.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:24:56.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:24:56.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:24:56.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:24:56.414 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:24:56.415 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:24:56.416 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:24:56.728 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:24:56.729 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:56.729 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:56.729 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:56.729 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:56.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[100622]: from='client.64331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[100622]: from='client.54375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3110089346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:56.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[102844]: from='client.64331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[102844]: from='client.54375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3110089346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:56.766 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:56 smithi196 ceph-mon[76349]: from='client.64331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:56 smithi196 ceph-mon[76349]: from='client.54375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:56.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3110089346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[102844]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[102844]: from='client.54393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[102844]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3021193825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[100622]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[100622]: from='client.54393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[100622]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3021193825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:57 smithi196 ceph-mon[76349]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:57 smithi196 ceph-mon[76349]: from='client.54393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:57 smithi196 ceph-mon[76349]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3021193825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:59 smithi184 ceph-mon[100622]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:59 smithi184 ceph-mon[102844]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:59 smithi196 ceph-mon[76349]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:00.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:00.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:00.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[100622]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[102844]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:01 smithi196 ceph-mon[76349]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:03 smithi196 ceph-mon[76349]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[100622]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:03 smithi184 ceph-mon[102844]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:05 smithi196 ceph-mon[76349]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:05 smithi184 ceph-mon[100622]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:05 smithi184 ceph-mon[102844]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:07 smithi184 ceph-mon[100622]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:07 smithi184 ceph-mon[102844]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:07 smithi196 ceph-mon[76349]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:09 smithi184 ceph-mon[100622]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:09 smithi184 ceph-mon[102844]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:09 smithi196 ceph-mon[76349]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:10.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:10.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:10.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:11 smithi184 ceph-mon[100622]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:11 smithi184 ceph-mon[102844]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:11 smithi196 ceph-mon[76349]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:25:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:13 smithi184 ceph-mon[100622]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:13 smithi184 ceph-mon[102844]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:13 smithi196 ceph-mon[76349]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:15.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:15 smithi184 ceph-mon[100622]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:15.740 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:15 smithi184 ceph-mon[102844]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:15 smithi196 ceph-mon[76349]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:17 smithi184 ceph-mon[100622]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:17 smithi184 ceph-mon[102844]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:17.858 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:17 smithi196 ceph-mon[76349]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:19 smithi184 ceph-mon[100622]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:19 smithi184 ceph-mon[102844]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:19 smithi196 ceph-mon[76349]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:20.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:20.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:20.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[100622]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[102844]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:21 smithi196 ceph-mon[76349]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:22.286 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:25:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[100622]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[102844]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:23 smithi196 ceph-mon[76349]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:25:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:25 smithi184 ceph-mon[100622]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:25 smithi184 ceph-mon[102844]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:25 smithi196 ceph-mon[76349]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:27.022 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 2h - - 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 6s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 6s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:25:27.558 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 6s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 6s ago 2h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 6s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 6s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:25:27.559 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:25:27.560 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:25:27.560 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:25:27.560 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:25:27.560 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:25:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:27 smithi184 ceph-mon[100622]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:27 smithi184 ceph-mon[102844]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:25:27.873 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:25:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:27 smithi196 ceph-mon[76349]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:28.143 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:25:28.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:25:28.456 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:25:28.456 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:28.456 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:28.456 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:28.456 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[102844]: from='client.54405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[102844]: from='client.64373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[102844]: from='client.64379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1767792189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1133590197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[100622]: from='client.54405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[100622]: from='client.64373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[100622]: from='client.64379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1767792189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1133590197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:28.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:28 smithi196 ceph-mon[76349]: from='client.54405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:28 smithi196 ceph-mon[76349]: from='client.64373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:28 smithi196 ceph-mon[76349]: from='client.64379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:28.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1767792189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:28.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1133590197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[102844]: from='client.64391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[102844]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[100622]: from='client.64391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[100622]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:29 smithi196 ceph-mon[76349]: from='client.64391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:29 smithi196 ceph-mon[76349]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:30.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:30.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:30.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:31 smithi184 ceph-mon[100622]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:31 smithi184 ceph-mon[102844]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:31 smithi196 ceph-mon[76349]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:25:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:33 smithi184 ceph-mon[100622]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:33 smithi184 ceph-mon[102844]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:33 smithi196 ceph-mon[76349]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:35 smithi184 ceph-mon[100622]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:35 smithi184 ceph-mon[102844]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:35 smithi196 ceph-mon[76349]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:37.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:37 smithi196 ceph-mon[76349]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:37 smithi184 ceph-mon[100622]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:37 smithi184 ceph-mon[102844]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:39 smithi184 ceph-mon[100622]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:39 smithi184 ceph-mon[102844]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:39.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:39 smithi196 ceph-mon[76349]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:40.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:40.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:40.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[100622]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[102844]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:41 smithi196 ceph-mon[76349]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:25:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:43 smithi196 ceph-mon[76349]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[100622]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[102844]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:25:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:45 smithi196 ceph-mon[76349]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:45 smithi184 ceph-mon[100622]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:45 smithi184 ceph-mon[102844]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:47 smithi196 ceph-mon[76349]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:48.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:47 smithi184 ceph-mon[100622]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:48.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:47 smithi184 ceph-mon[102844]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:49 smithi196 ceph-mon[76349]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:49 smithi184 ceph-mon[100622]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:49 smithi184 ceph-mon[102844]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:50.765Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:50.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:25:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:25:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:25:50.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:25:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:51 smithi196 ceph-mon[76349]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:51 smithi184 ceph-mon[100622]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:51 smithi184 ceph-mon[102844]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:25:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:25:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:53 smithi196 ceph-mon[76349]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:53 smithi184 ceph-mon[100622]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:53 smithi184 ceph-mon[102844]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:55 smithi184 ceph-mon[100622]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:55 smithi184 ceph-mon[102844]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:55 smithi196 ceph-mon[76349]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:57.947 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:57 smithi196 ceph-mon[76349]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:57 smithi184 ceph-mon[100622]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:57 smithi184 ceph-mon[102844]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:58.747 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:25:59.278 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:25:59.278 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 2h - - 2024-09-20T22:25:59.278 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 2h - - 2024-09-20T22:25:59.278 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:25:59.278 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 16s ago 2h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 16s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 18s ago 2h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 16s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 18s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 16s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 18s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 18s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 18s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:25:59.279 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 16s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 18s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:25:59.280 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:25:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:25:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:25:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:25:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:25:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:25:59.590 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:25:59.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:59 smithi196 ceph-mon[76349]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:59.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:59 smithi196 ceph-mon[76349]: from='client.64403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:59.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:59.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:25:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1848525778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:25:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:25:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:25:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:25:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:25:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:25:59.862 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:25:59.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[102844]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:59.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[102844]: from='client.64403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:59.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:59.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1848525778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:59.862 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[100622]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:25:59.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[100622]: from='client.64403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:59.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:59.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:25:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1848525778' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:00.173 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:26:00.174 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:00.174 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:00.174 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:00.174 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:00.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:00 smithi196 ceph-mon[76349]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:00.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:00 smithi196 ceph-mon[76349]: from='client.64415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:00.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4029662899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[100622]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[100622]: from='client.64415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4029662899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[102844]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[102844]: from='client.64415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4029662899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:00.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:00.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:00.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:01.904 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:01 smithi196 ceph-mon[76349]: from='client.64427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:01.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:01 smithi196 ceph-mon[76349]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:01.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:01.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:01.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[100622]: from='client.64427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[100622]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[102844]: from='client.64427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[102844]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[100622]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[102844]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:03 smithi196 ceph-mon[76349]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:05 smithi184 ceph-mon[100622]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:05 smithi184 ceph-mon[102844]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:05 smithi196 ceph-mon[76349]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:06 smithi184 ceph-mon[100622]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:06 smithi184 ceph-mon[102844]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:06 smithi196 ceph-mon[76349]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:09 smithi184 ceph-mon[100622]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:09 smithi184 ceph-mon[102844]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:09 smithi196 ceph-mon[76349]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:10.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:10.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:10.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:11 smithi184 ceph-mon[100622]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:11 smithi184 ceph-mon[102844]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:11 smithi196 ceph-mon[76349]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:26:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:13 smithi184 ceph-mon[102844]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:13 smithi184 ceph-mon[100622]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:13 smithi196 ceph-mon[76349]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:15 smithi196 ceph-mon[76349]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:15 smithi184 ceph-mon[100622]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:15 smithi184 ceph-mon[102844]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:17.857 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:17 smithi184 ceph-mon[100622]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:17.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:17 smithi184 ceph-mon[102844]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:17 smithi196 ceph-mon[76349]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:19 smithi196 ceph-mon[76349]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:19 smithi184 ceph-mon[100622]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:19 smithi184 ceph-mon[102844]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:20.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:20.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:20.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:21 smithi196 ceph-mon[76349]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[100622]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[102844]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:26:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:23 smithi196 ceph-mon[76349]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[100622]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[102844]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:25 smithi196 ceph-mon[76349]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:25 smithi184 ceph-mon[100622]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:25 smithi184 ceph-mon[102844]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:27 smithi196 ceph-mon[76349]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:27 smithi184 ceph-mon[100622]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:27 smithi184 ceph-mon[102844]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:29 smithi184 ceph-mon[100622]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:29 smithi184 ceph-mon[102844]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:29 smithi196 ceph-mon[76349]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:26:30.998 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 2h - - 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 2h - - 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 9s ago 2h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 8s ago 2h 90.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 9s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 9s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 9s ago 2h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 8s ago 2h 195M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 9s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:26:30.999 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 9s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 8s ago 2h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 9s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 9s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 9s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 9s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 8s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 8s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 9s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 9s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:26:31.000 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:26:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:30.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:30.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:30.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:31.315 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:26:31.584 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:26:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:31 smithi196 ceph-mon[76349]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:31 smithi196 ceph-mon[76349]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:31 smithi196 ceph-mon[76349]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/453951327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:31.894 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:26:31.894 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:31.894 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:31.895 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:31.895 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[100622]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[100622]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[100622]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/453951327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[102844]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[102844]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:31.895 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[102844]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:31.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/453951327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:26:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:32 smithi196 ceph-mon[76349]: from='client.54489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:32 smithi196 ceph-mon[76349]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1486288221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[102844]: from='client.54489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[102844]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1486288221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[100622]: from='client.54489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[100622]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:32.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1486288221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:33 smithi184 ceph-mon[100622]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:33 smithi184 ceph-mon[102844]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:33 smithi196 ceph-mon[76349]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:35 smithi196 ceph-mon[76349]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:35 smithi184 ceph-mon[100622]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:35 smithi184 ceph-mon[102844]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:37 smithi196 ceph-mon[76349]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:37 smithi184 ceph-mon[100622]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:37 smithi184 ceph-mon[102844]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:39 smithi196 ceph-mon[76349]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:39 smithi184 ceph-mon[100622]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:39 smithi184 ceph-mon[102844]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:40.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:40.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:40.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:41 smithi196 ceph-mon[76349]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:41.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[100622]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[102844]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:26:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:43 smithi196 ceph-mon[76349]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[102844]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[100622]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:26:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:45 smithi196 ceph-mon[76349]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:45 smithi184 ceph-mon[100622]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:45 smithi184 ceph-mon[102844]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:47 smithi196 ceph-mon[76349]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:47 smithi184 ceph-mon[100622]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:47 smithi184 ceph-mon[102844]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:49 smithi196 ceph-mon[76349]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:49 smithi184 ceph-mon[100622]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:49 smithi184 ceph-mon[102844]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:50.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:50.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:26:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:26:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:26:50.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:26:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:51 smithi196 ceph-mon[76349]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:51 smithi184 ceph-mon[102844]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:51 smithi184 ceph-mon[100622]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:26:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:26:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:53 smithi184 ceph-mon[100622]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:53 smithi184 ceph-mon[102844]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:53 smithi196 ceph-mon[76349]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:55 smithi184 ceph-mon[102844]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:55 smithi184 ceph-mon[100622]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:55 smithi196 ceph-mon[76349]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:57 smithi184 ceph-mon[100622]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:57 smithi184 ceph-mon[102844]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:57 smithi196 ceph-mon[76349]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:59 smithi184 ceph-mon[100622]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:59 smithi184 ceph-mon[102844]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:59 smithi196 ceph-mon[76349]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:00.992 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:00.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:00.992 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:00.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:00.992 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:00.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[102844]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[100622]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:01 smithi196 ceph-mon[76349]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:01.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:01.826 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:02.186 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:27:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:27:02.715 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:27:02.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T22:27:02.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 2h - - 2024-09-20T22:27:02.715 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 19s ago 2h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 2h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 19s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 19s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:27:02.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:27:02.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:27:03.027 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:27:03.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:03.029 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:27:03.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:27:03.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:27:03.029 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:27:03.303 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:27:03.304 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:27:03.622 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:27:03.622 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:03.622 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:03.623 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:03.623 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='client.64475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/674677303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2569864036' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='client.64475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/674677303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2569864036' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='client.64475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/674677303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:04.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2569864036' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:04 smithi184 ceph-mon[100622]: from='client.64499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:04 smithi184 ceph-mon[102844]: from='client.64499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:04 smithi196 ceph-mon[76349]: from='client.64499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:05 smithi184 ceph-mon[100622]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:05 smithi184 ceph-mon[102844]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:05 smithi196 ceph-mon[76349]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:07 smithi184 ceph-mon[100622]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:07 smithi184 ceph-mon[102844]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:07 smithi196 ceph-mon[76349]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:09 smithi184 ceph-mon[100622]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:09 smithi184 ceph-mon[102844]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:09 smithi196 ceph-mon[76349]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:10.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:10.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:10.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:11 smithi184 ceph-mon[100622]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:11 smithi184 ceph-mon[102844]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:11 smithi196 ceph-mon[76349]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:27:13.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:13 smithi196 ceph-mon[76349]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:13 smithi184 ceph-mon[100622]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:13 smithi184 ceph-mon[102844]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:15.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:15.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:15 smithi184 ceph-mon[100622]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:15 smithi184 ceph-mon[102844]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:15 smithi196 ceph-mon[76349]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:17 smithi184 ceph-mon[100622]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:27:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:17 smithi184 ceph-mon[102844]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:27:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:17 smithi196 ceph-mon[76349]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:27:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:19 smithi184 ceph-mon[100622]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:19 smithi184 ceph-mon[102844]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:19 smithi196 ceph-mon[76349]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:20.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:20.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:20.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:21.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:21 smithi196 ceph-mon[76349]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:21.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:21.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:21.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[100622]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[102844]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:27:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:23 smithi184 ceph-mon[100622]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:23 smithi184 ceph-mon[102844]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:23 smithi196 ceph-mon[76349]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:25 smithi184 ceph-mon[100622]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:25 smithi184 ceph-mon[102844]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:25 smithi196 ceph-mon[76349]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:27 smithi196 ceph-mon[76349]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:27 smithi184 ceph-mon[100622]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:27 smithi184 ceph-mon[102844]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:29 smithi196 ceph-mon[76349]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:29 smithi184 ceph-mon[100622]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:29 smithi184 ceph-mon[102844]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:30.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:30.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:30.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:31 smithi196 ceph-mon[76349]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:31 smithi184 ceph-mon[100622]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:31 smithi184 ceph-mon[102844]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:27:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:33 smithi184 ceph-mon[100622]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:33 smithi184 ceph-mon[102844]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:33 smithi196 ceph-mon[76349]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:33.908 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 2h - - 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:27:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 12s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 2h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 11s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 12s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 12s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 12s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 12s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 11s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 11s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:27:34.440 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 12s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:27:34.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:27:34.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 12s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:27:34.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:27:34.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:27:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:27:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:27:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:27:34.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:27:34.754 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:27:34.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:34.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:35.020 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:27:35.021 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:27:35.336 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:27:35.336 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:35.336 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:35.336 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:35.336 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: from='client.64511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: from='client.64517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: from='client.64523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4132729280' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:35.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1725467305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: from='client.64511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: from='client.64517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: from='client.64523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4132729280' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1725467305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: from='client.64511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: from='client.64517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: from='client.64523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4132729280' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1725467305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:36 smithi196 ceph-mon[76349]: from='client.64535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:36 smithi184 ceph-mon[100622]: from='client.64535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:36 smithi184 ceph-mon[102844]: from='client.64535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:37 smithi196 ceph-mon[76349]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:37 smithi184 ceph-mon[100622]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:37 smithi184 ceph-mon[102844]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:39.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:39 smithi184 ceph-mon[100622]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:39.763 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:39 smithi184 ceph-mon[102844]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:39 smithi196 ceph-mon[76349]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:40.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:40.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:40.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:40.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:41.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:41 smithi184 ceph-mon[100622]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:41.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:41 smithi184 ceph-mon[102844]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:41.823 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:41 smithi196 ceph-mon[76349]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:27:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[100622]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[102844]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:43 smithi196 ceph-mon[76349]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:27:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:45 smithi184 ceph-mon[100622]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:45 smithi184 ceph-mon[102844]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:45 smithi196 ceph-mon[76349]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:47 smithi184 ceph-mon[100622]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:47 smithi184 ceph-mon[102844]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:47 smithi196 ceph-mon[76349]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:49 smithi184 ceph-mon[100622]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:49 smithi184 ceph-mon[102844]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:49 smithi196 ceph-mon[76349]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:50.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:50.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:27:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:27:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:27:50.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:27:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:51 smithi184 ceph-mon[100622]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:51 smithi184 ceph-mon[102844]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:51 smithi196 ceph-mon[76349]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:27:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T22:27:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:53 smithi184 ceph-mon[100622]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:53 smithi184 ceph-mon[102844]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:53 smithi196 ceph-mon[76349]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:55 smithi184 ceph-mon[100622]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:55 smithi184 ceph-mon[102844]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:55 smithi196 ceph-mon[76349]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:56 smithi196 ceph-mon[76349]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:56 smithi184 ceph-mon[100622]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:56 smithi184 ceph-mon[102844]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:59 smithi196 ceph-mon[76349]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:59 smithi184 ceph-mon[102844]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:59 smithi184 ceph-mon[100622]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:00.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:00.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:00.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:01 smithi196 ceph-mon[76349]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:01 smithi184 ceph-mon[100622]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:01 smithi184 ceph-mon[102844]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:28:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[100622]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:02 smithi184 ceph-mon[102844]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:02 smithi196 ceph-mon[76349]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:04.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:05.627 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:28:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:05 smithi184 ceph-mon[102844]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:05 smithi184 ceph-mon[100622]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:05 smithi196 ceph-mon[76349]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:06.154 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:28:06.154 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 2h - - 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 2h - - 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 2s ago 2h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 2s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 2h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 2s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 4s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:28:06.155 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 4s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 4s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 2s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 4s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:28:06.156 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:28:06.463 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:28:06.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:28:06.465 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:28:06.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:28:06.465 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:28:06.731 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:28:06.731 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:28:06.731 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:28:06.732 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:28:06.733 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:28:06.733 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:28:06.733 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:28:06.733 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:28:06.733 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:28:06.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:06 smithi184 ceph-mon[100622]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:06.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2730028970' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:06.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:06 smithi184 ceph-mon[102844]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:06.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2730028970' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:06 smithi196 ceph-mon[76349]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2730028970' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:07.042 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:28:07.042 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:07.042 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:07.042 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:07.042 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[100622]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[100622]: from='client.64559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[100622]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[100622]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/435402084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[102844]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[102844]: from='client.64559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[102844]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[102844]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/435402084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:07 smithi196 ceph-mon[76349]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:07 smithi196 ceph-mon[76349]: from='client.64559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:07 smithi196 ceph-mon[76349]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:07 smithi196 ceph-mon[76349]: from='client.54609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/435402084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:09 smithi196 ceph-mon[76349]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:09 smithi184 ceph-mon[100622]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:09 smithi184 ceph-mon[102844]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:10.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:10.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:10.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:11 smithi196 ceph-mon[76349]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:11 smithi184 ceph-mon[100622]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:11 smithi184 ceph-mon[102844]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:28:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:13 smithi196 ceph-mon[76349]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:13.890 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:28:13.812807592Z level=info msg="Completed cleanup jobs" duration=12.951482ms 2024-09-20T22:28:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:13 smithi184 ceph-mon[102844]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:13 smithi184 ceph-mon[100622]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:14.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:14.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:14.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:14.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:15 smithi196 ceph-mon[76349]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:15 smithi184 ceph-mon[102844]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:15 smithi184 ceph-mon[100622]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:17 smithi196 ceph-mon[76349]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:17 smithi184 ceph-mon[100622]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:17 smithi184 ceph-mon[102844]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:19 smithi196 ceph-mon[76349]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:19 smithi184 ceph-mon[100622]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:19 smithi184 ceph-mon[102844]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:20.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:20.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:20.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:21 smithi196 ceph-mon[76349]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:21.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:21 smithi184 ceph-mon[100622]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:21.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:21 smithi184 ceph-mon[102844]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:22.253 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:28:23.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:22 smithi196 ceph-mon[76349]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[100622]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:23.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:22 smithi184 ceph-mon[102844]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:24.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:24.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:24.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:25 smithi184 ceph-mon[100622]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:25 smithi184 ceph-mon[102844]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:25 smithi196 ceph-mon[76349]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:27 smithi184 ceph-mon[100622]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:27 smithi184 ceph-mon[102844]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:27 smithi196 ceph-mon[76349]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:29 smithi184 ceph-mon[100622]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:29 smithi184 ceph-mon[102844]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:29 smithi196 ceph-mon[76349]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:30.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:30.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:30.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:31 smithi184 ceph-mon[100622]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:31 smithi184 ceph-mon[102844]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:31 smithi196 ceph-mon[76349]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:28:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:33 smithi184 ceph-mon[100622]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:33 smithi184 ceph-mon[102844]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:33 smithi196 ceph-mon[76349]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:35 smithi184 ceph-mon[102844]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:35 smithi184 ceph-mon[100622]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:35 smithi196 ceph-mon[76349]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:37.336 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:28:37.866 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 15s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 15s ago 2h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 15s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 15s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 15s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 15s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:28:37.867 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 15s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:28:37.868 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:28:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:37 smithi184 ceph-mon[100622]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:37 smithi184 ceph-mon[102844]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:37 smithi196 ceph-mon[76349]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:28:38.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:28:38.177 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:28:38.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:28:38.446 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:28:38.446 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:28:38.446 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:28:38.446 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:28:38.446 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:28:38.757 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:28:38.757 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:38.757 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:38.757 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:38.757 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:38.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[100622]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:38.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[100622]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:38.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3765205863' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:38.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[102844]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:38.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[102844]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:38.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3765205863' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:38 smithi196 ceph-mon[76349]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:38 smithi196 ceph-mon[76349]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3765205863' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[100622]: from='client.64595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[100622]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[100622]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3197454648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[102844]: from='client.64595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[102844]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[102844]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3197454648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:39 smithi196 ceph-mon[76349]: from='client.64595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:39 smithi196 ceph-mon[76349]: from='client.54645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:39 smithi196 ceph-mon[76349]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3197454648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:40.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:40.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:40.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:41 smithi184 ceph-mon[100622]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:41 smithi184 ceph-mon[102844]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:28:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:41 smithi196 ceph-mon[76349]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[100622]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:43 smithi184 ceph-mon[102844]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:43 smithi196 ceph-mon[76349]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:44.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:44.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:44.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:44.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:44.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:44.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:28:44.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:45 smithi184 ceph-mon[100622]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:45 smithi184 ceph-mon[102844]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:45 smithi196 ceph-mon[76349]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:47 smithi196 ceph-mon[76349]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:47 smithi184 ceph-mon[100622]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:47 smithi184 ceph-mon[102844]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:49 smithi196 ceph-mon[76349]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:49 smithi184 ceph-mon[100622]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:49 smithi184 ceph-mon[102844]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:50.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:50.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:28:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:28:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:28:50.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:28:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:51 smithi196 ceph-mon[76349]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:51 smithi184 ceph-mon[100622]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:51 smithi184 ceph-mon[102844]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:28:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:28:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:53 smithi196 ceph-mon[76349]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:53 smithi184 ceph-mon[100622]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:53 smithi184 ceph-mon[102844]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:55 smithi196 ceph-mon[76349]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:55 smithi184 ceph-mon[100622]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:55 smithi184 ceph-mon[102844]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:57 smithi196 ceph-mon[76349]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:57 smithi184 ceph-mon[100622]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:57 smithi184 ceph-mon[102844]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:59 smithi196 ceph-mon[76349]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:59 smithi184 ceph-mon[100622]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:59 smithi184 ceph-mon[102844]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:00.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:00.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:00.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:01 smithi196 ceph-mon[76349]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:01 smithi184 ceph-mon[100622]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:01 smithi184 ceph-mon[102844]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:29:03.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:03 smithi196 ceph-mon[76349]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[100622]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:03 smithi184 ceph-mon[102844]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:04.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:04.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:05.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:05 smithi184 ceph-mon[100622]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:05.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:05 smithi196 ceph-mon[76349]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:05.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:05 smithi184 ceph-mon[102844]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:07 smithi196 ceph-mon[76349]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:07 smithi184 ceph-mon[100622]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:07 smithi184 ceph-mon[102844]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.052 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:29:09.579 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 2h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:29:09.603 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 7s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 7s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 7s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 7s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:29:09.604 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:29:09.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:09 smithi184 ceph-mon[100622]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:09 smithi196 ceph-mon[76349]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.890 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:29:09.890 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:29:09.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:29:09.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:29:09.891 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:29:09.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:09 smithi184 ceph-mon[102844]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:10.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:29:10.159 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:29:10.159 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:29:10.159 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:29:10.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:29:10.469 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:29:10.469 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:10.470 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:10.470 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:10.470 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:10.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[100622]: from='client.64619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[100622]: from='client.64625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[100622]: from='client.54669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3020007407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:10.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3282463764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:10.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[102844]: from='client.64619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[102844]: from='client.64625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[102844]: from='client.54669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3020007407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:10.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3282463764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:10 smithi196 ceph-mon[76349]: from='client.64619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:10 smithi196 ceph-mon[76349]: from='client.64625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:10 smithi196 ceph-mon[76349]: from='client.54669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3020007407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3282463764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:10.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:10.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:10.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:11 smithi196 ceph-mon[76349]: from='client.64643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:11 smithi196 ceph-mon[76349]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:11 smithi184 ceph-mon[100622]: from='client.64643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:11 smithi184 ceph-mon[100622]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:11 smithi184 ceph-mon[102844]: from='client.64643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:11 smithi184 ceph-mon[102844]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:29:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:13 smithi196 ceph-mon[76349]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:13 smithi184 ceph-mon[100622]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:13 smithi184 ceph-mon[102844]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:14.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:14.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:15 smithi196 ceph-mon[76349]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:15 smithi184 ceph-mon[100622]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:15 smithi184 ceph-mon[102844]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:17 smithi196 ceph-mon[76349]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:17 smithi184 ceph-mon[100622]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:18.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:17 smithi184 ceph-mon[102844]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:19 smithi196 ceph-mon[76349]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:19 smithi184 ceph-mon[100622]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:19 smithi184 ceph-mon[102844]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:20.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:20.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:20.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:21 smithi196 ceph-mon[76349]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:21 smithi184 ceph-mon[100622]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:21 smithi184 ceph-mon[102844]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:29:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:23 smithi196 ceph-mon[76349]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[100622]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:23.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:23 smithi184 ceph-mon[102844]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:24.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:24.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:24.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:24.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:25.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:25 smithi196 ceph-mon[76349]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:25.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:25 smithi184 ceph-mon[100622]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:25.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:25 smithi184 ceph-mon[102844]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:27 smithi196 ceph-mon[76349]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:27 smithi184 ceph-mon[100622]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:27 smithi184 ceph-mon[102844]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:29 smithi196 ceph-mon[76349]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:29 smithi184 ceph-mon[100622]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:29 smithi184 ceph-mon[102844]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:30.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:30.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:30.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:31 smithi196 ceph-mon[76349]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:31 smithi184 ceph-mon[100622]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:31 smithi184 ceph-mon[102844]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:29:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:33 smithi184 ceph-mon[100622]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:33 smithi184 ceph-mon[102844]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:33 smithi196 ceph-mon[76349]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:35 smithi196 ceph-mon[76349]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:35 smithi184 ceph-mon[100622]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:35 smithi184 ceph-mon[102844]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:37 smithi196 ceph-mon[76349]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:37 smithi184 ceph-mon[100622]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:37 smithi184 ceph-mon[102844]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:39 smithi196 ceph-mon[76349]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:39 smithi184 ceph-mon[100622]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:39 smithi184 ceph-mon[102844]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:40.764 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:29:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:40.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:40.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:40.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:41.289 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 2h - - 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 18s ago 2h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 18s ago 2h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:29:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 18s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 18s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 18s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 17s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 17s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 18s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 17s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:29:41.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 17s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:29:41.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:29:41.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:29:41.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:29:41.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:29:41.604 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:29:41.880 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:29:41.881 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:29:41.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[102844]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:41.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[102844]: from='client.64655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:41.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3739545686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:41.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[100622]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:41.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[100622]: from='client.64655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:41.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3739545686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:41 smithi196 ceph-mon[76349]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:41 smithi196 ceph-mon[76349]: from='client.64655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3739545686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:42.189 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:29:42.190 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:42.190 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:42.190 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:42.190 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:42.190 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:29:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='client.64661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='client.64667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/94136091' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='client.64661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='client.64667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/94136091' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='client.64661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='client.64667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/94136091' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:42.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:43.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:43 smithi184 ceph-mon[100622]: from='client.54717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:43.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:43 smithi184 ceph-mon[100622]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:43.840 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:43 smithi184 ceph-mon[102844]: from='client.54717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:43.840 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:43 smithi184 ceph-mon[102844]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:43 smithi196 ceph-mon[76349]: from='client.54717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:43 smithi196 ceph-mon[76349]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:44.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:44.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:29:45.813 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:45 smithi184 ceph-mon[102844]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:45.814 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:45 smithi184 ceph-mon[100622]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:45.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:45 smithi196 ceph-mon[76349]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:47 smithi184 ceph-mon[100622]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:47 smithi184 ceph-mon[102844]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:47 smithi196 ceph-mon[76349]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:49 smithi184 ceph-mon[100622]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:49 smithi184 ceph-mon[102844]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:49 smithi196 ceph-mon[76349]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:50.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:50.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:29:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:29:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:29:50.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:29:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:51 smithi184 ceph-mon[100622]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:51 smithi184 ceph-mon[102844]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:51 smithi196 ceph-mon[76349]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:29:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:29:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:53 smithi184 ceph-mon[100622]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:53 smithi184 ceph-mon[102844]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:53 smithi196 ceph-mon[76349]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:55 smithi184 ceph-mon[100622]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:55 smithi184 ceph-mon[102844]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:55 smithi196 ceph-mon[76349]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:57 smithi184 ceph-mon[102844]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:57 smithi184 ceph-mon[100622]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:57 smithi196 ceph-mon[76349]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:59 smithi184 ceph-mon[100622]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:59 smithi184 ceph-mon[102844]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:00.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:59 smithi196 ceph-mon[76349]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:00.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:00.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:00.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:01 smithi184 ceph-mon[100622]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:01 smithi184 ceph-mon[102844]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:01 smithi196 ceph-mon[76349]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:30:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[100622]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[102844]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:03 smithi196 ceph-mon[76349]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:04.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:04.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:05.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:05 smithi184 ceph-mon[100622]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:05.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:05 smithi184 ceph-mon[102844]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:05 smithi196 ceph-mon[76349]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:07 smithi196 ceph-mon[76349]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:07 smithi184 ceph-mon[100622]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:07 smithi184 ceph-mon[102844]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:09 smithi196 ceph-mon[76349]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:09 smithi184 ceph-mon[100622]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:09 smithi184 ceph-mon[102844]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:10.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:10.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:10.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:11 smithi196 ceph-mon[76349]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:11 smithi184 ceph-mon[100622]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:11 smithi184 ceph-mon[102844]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:30:12.488 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 2h - - 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 2h - - 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 8s ago 2h 92.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:30:13.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 2h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 8s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 2h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:30:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 8s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 10s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 10s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 8s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:30:13.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:30:13.043 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 8s ago 2h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:30:13.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:30:13.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:30:13.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:30:13.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:13.352 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:30:13.353 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:30:13.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:30:13.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:30:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:13 smithi196 ceph-mon[76349]: from='client.54729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:13 smithi196 ceph-mon[76349]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:13 smithi196 ceph-mon[76349]: from='client.64697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1083032957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:13.953 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:13.953 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:13.953 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:13.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[102844]: from='client.54729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[102844]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[102844]: from='client.64697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1083032957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[100622]: from='client.54729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[100622]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[100622]: from='client.64697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:13.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1083032957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:14 smithi196 ceph-mon[76349]: from='client.64703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:14 smithi196 ceph-mon[76349]: from='client.54753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1777976975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[102844]: from='client.64703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[102844]: from='client.54753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1777976975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[100622]: from='client.64703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[100622]: from='client.54753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1777976975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:15 smithi184 ceph-mon[100622]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:15 smithi184 ceph-mon[102844]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:15 smithi196 ceph-mon[76349]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:17 smithi184 ceph-mon[100622]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:17 smithi184 ceph-mon[102844]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:17 smithi196 ceph-mon[76349]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:19 smithi184 ceph-mon[100622]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:19 smithi184 ceph-mon[102844]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:19 smithi196 ceph-mon[76349]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:20.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:20.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:20.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:21 smithi184 ceph-mon[100622]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:21 smithi184 ceph-mon[102844]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:21 smithi196 ceph-mon[76349]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:30:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[102844]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[100622]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:23 smithi196 ceph-mon[76349]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:25.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:25.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:26.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:25 smithi184 ceph-mon[100622]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:26.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:25 smithi184 ceph-mon[102844]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:26.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:25 smithi196 ceph-mon[76349]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:27 smithi196 ceph-mon[76349]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:28.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:27 smithi184 ceph-mon[102844]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:28.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:27 smithi184 ceph-mon[100622]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:29 smithi184 ceph-mon[100622]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:29 smithi184 ceph-mon[102844]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:29 smithi196 ceph-mon[76349]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:30.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:30.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:30.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:31 smithi184 ceph-mon[100622]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:31 smithi184 ceph-mon[102844]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:30:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:31 smithi196 ceph-mon[76349]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:33 smithi184 ceph-mon[100622]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:33 smithi184 ceph-mon[102844]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:33 smithi196 ceph-mon[76349]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:35 smithi184 ceph-mon[100622]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:35 smithi184 ceph-mon[102844]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:35 smithi196 ceph-mon[76349]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:37 smithi184 ceph-mon[100622]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:37 smithi184 ceph-mon[102844]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:37 smithi196 ceph-mon[76349]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:39 smithi184 ceph-mon[100622]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:39 smithi184 ceph-mon[102844]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:39 smithi196 ceph-mon[76349]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:40.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:40.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:40.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:41 smithi184 ceph-mon[100622]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:30:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:41 smithi184 ceph-mon[102844]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:41 smithi196 ceph-mon[76349]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[100622]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[102844]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:43 smithi196 ceph-mon[76349]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.248 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 2h - - 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 0s ago 2h 90.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:30:44.781 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 2h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 2h 202M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 2h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 0s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 0s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 0s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 2h 78.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:30:44.782 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:30:44.783 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 0s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:30:44.783 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:30:44.783 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 0s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:30:45.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:30:45.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:30:45.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:30:45.094 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:30:45.394 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:30:45.394 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:30:45.395 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:30:45.716 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:45.717 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:45.717 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:45.717 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:45.717 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.64727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.64739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2510600018' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.54789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3914501194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.64727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.64739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2510600018' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.54789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3914501194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:46.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.64727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:46.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.64739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2510600018' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:46.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.54789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:46.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3914501194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:47.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:46 smithi196 ceph-mon[76349]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:47.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:46 smithi184 ceph-mon[102844]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:47.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:46 smithi184 ceph-mon[100622]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:49 smithi196 ceph-mon[76349]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:49 smithi184 ceph-mon[100622]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:49 smithi184 ceph-mon[102844]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:50.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:50.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:30:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:30:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:30:50.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:30:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:51 smithi196 ceph-mon[76349]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:51 smithi184 ceph-mon[100622]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:51 smithi184 ceph-mon[102844]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:30:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:30:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:53 smithi196 ceph-mon[76349]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:53 smithi184 ceph-mon[100622]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:53 smithi184 ceph-mon[102844]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:55 smithi184 ceph-mon[100622]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:55 smithi184 ceph-mon[102844]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:55 smithi196 ceph-mon[76349]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:57 smithi184 ceph-mon[100622]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:57 smithi184 ceph-mon[102844]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:57 smithi196 ceph-mon[76349]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:59 smithi184 ceph-mon[100622]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:59 smithi184 ceph-mon[102844]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:59 smithi196 ceph-mon[76349]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:00.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:00.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:00.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:01 smithi184 ceph-mon[100622]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:01 smithi184 ceph-mon[102844]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:01 smithi196 ceph-mon[76349]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:31:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[100622]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[102844]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:03 smithi196 ceph-mon[76349]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[100622]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[102844]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:05 smithi196 ceph-mon[76349]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:07 smithi184 ceph-mon[100622]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:07 smithi184 ceph-mon[102844]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:08.070 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:07 smithi196 ceph-mon[76349]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:09 smithi196 ceph-mon[76349]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:09 smithi184 ceph-mon[100622]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:09 smithi184 ceph-mon[102844]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:10.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:10.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:10.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:11 smithi184 ceph-mon[100622]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:11 smithi184 ceph-mon[102844]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:11 smithi196 ceph-mon[76349]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:31:13.960 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:13 smithi196 ceph-mon[76349]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:13 smithi184 ceph-mon[100622]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:13 smithi184 ceph-mon[102844]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:15.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:15.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:16.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:15 smithi184 ceph-mon[100622]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:16.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:15 smithi184 ceph-mon[102844]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:16.012 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:31:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:15 smithi196 ceph-mon[76349]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 2h - - 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 2h - - 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 13s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 13s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 13s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 13s ago 2h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 11s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 13s ago 2h 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:31:16.537 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 13s ago 2h 16.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 13s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 13s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 11s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 11s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 78.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:31:16.538 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:31:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:31:16.849 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:31:17.430 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:31:17.430 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:17.430 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:17.430 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:17.430 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: from='client.64775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2164798744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/936858696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: from='client.64775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2164798744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/936858696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: from='client.64775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2164798744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/936858696' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:19.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:18 smithi184 ceph-mon[100622]: from='client.54825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:19.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:18 smithi184 ceph-mon[102844]: from='client.54825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:18 smithi196 ceph-mon[76349]: from='client.54825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:19 smithi184 ceph-mon[100622]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:19 smithi184 ceph-mon[102844]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:19 smithi196 ceph-mon[76349]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:20.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:20.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:20.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:21 smithi184 ceph-mon[100622]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:21 smithi184 ceph-mon[102844]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:21 smithi196 ceph-mon[76349]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[100622]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[102844]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:23 smithi196 ceph-mon[76349]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[100622]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[102844]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:25 smithi196 ceph-mon[76349]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:27 smithi184 ceph-mon[100622]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:27 smithi184 ceph-mon[102844]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:28.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:27 smithi196 ceph-mon[76349]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:29 smithi196 ceph-mon[76349]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:29 smithi184 ceph-mon[100622]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:29 smithi184 ceph-mon[102844]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:30.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:30.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:30.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:31 smithi184 ceph-mon[100622]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:31 smithi184 ceph-mon[102844]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:31 smithi196 ceph-mon[76349]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:31:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:33 smithi184 ceph-mon[100622]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:33 smithi184 ceph-mon[102844]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:33 smithi196 ceph-mon[76349]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:35 smithi184 ceph-mon[100622]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:35 smithi184 ceph-mon[102844]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:35 smithi196 ceph-mon[76349]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:37 smithi184 ceph-mon[100622]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:37 smithi184 ceph-mon[102844]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:37 smithi196 ceph-mon[76349]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:39 smithi184 ceph-mon[100622]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:39 smithi184 ceph-mon[102844]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:40.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:39 smithi196 ceph-mon[76349]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:40.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:40.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:40.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:41 smithi184 ceph-mon[100622]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:41 smithi184 ceph-mon[102844]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:31:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:41 smithi196 ceph-mon[76349]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[100622]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[102844]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:43 smithi196 ceph-mon[76349]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[100622]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[102844]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:45 smithi196 ceph-mon[76349]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:31:47.730 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:31:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:47 smithi184 ceph-mon[100622]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:47 smithi184 ceph-mon[102844]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:47 smithi196 ceph-mon[76349]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 2h - - 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 3s ago 2h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 2h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:31:48.270 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 3s ago 2h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 3s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 3s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 2h 74.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:31:48.271 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 3s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:31:48.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:31:48.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:31:48.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:31:48.588 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:31:48.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:48 smithi184 ceph-mon[100622]: from='client.64799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:48.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/77107124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:48.869 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:48 smithi184 ceph-mon[102844]: from='client.64799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:48.869 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/77107124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:49.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:48 smithi196 ceph-mon[76349]: from='client.64799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:49.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/77107124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:49.187 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:31:49.187 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:49.187 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:49.187 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:49.187 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[100622]: from='client.64805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[100622]: from='client.64811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[100622]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3291832060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[102844]: from='client.64805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[102844]: from='client.64811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[102844]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:50.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3291832060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:49 smithi196 ceph-mon[76349]: from='client.64805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:49 smithi196 ceph-mon[76349]: from='client.64811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:49 smithi196 ceph-mon[76349]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3291832060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:50 smithi184 ceph-mon[100622]: from='client.54861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:50 smithi184 ceph-mon[102844]: from='client.54861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:50.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:50.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:31:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:31:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:31:50.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:31:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:50 smithi196 ceph-mon[76349]: from='client.54861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:51 smithi184 ceph-mon[100622]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:52.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:51 smithi184 ceph-mon[102844]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:31:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:31:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:51 smithi196 ceph-mon[76349]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:52 smithi196 ceph-mon[76349]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:53.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:52 smithi184 ceph-mon[100622]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:53.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:52 smithi184 ceph-mon[102844]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:54 smithi196 ceph-mon[76349]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:54 smithi184 ceph-mon[100622]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:54 smithi184 ceph-mon[102844]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:57 smithi184 ceph-mon[102844]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:57 smithi184 ceph-mon[100622]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:57 smithi196 ceph-mon[76349]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:59 smithi184 ceph-mon[102844]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:59 smithi184 ceph-mon[100622]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:59 smithi196 ceph-mon[76349]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:00.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:00.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:00.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:01 smithi184 ceph-mon[100622]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:01 smithi184 ceph-mon[102844]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:01 smithi196 ceph-mon[76349]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:32:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[100622]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[102844]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:03 smithi196 ceph-mon[76349]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:05.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[100622]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[102844]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:05 smithi196 ceph-mon[76349]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:07 smithi184 ceph-mon[100622]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:07 smithi184 ceph-mon[102844]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:07 smithi196 ceph-mon[76349]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:09 smithi184 ceph-mon[100622]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:09 smithi184 ceph-mon[102844]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:10.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:09 smithi196 ceph-mon[76349]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:10.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:10.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:10.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:11 smithi196 ceph-mon[76349]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:11 smithi184 ceph-mon[100622]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:11 smithi184 ceph-mon[102844]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:32:14.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:13 smithi184 ceph-mon[100622]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:14.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:13 smithi184 ceph-mon[102844]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:13 smithi196 ceph-mon[76349]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:15 smithi184 ceph-mon[100622]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:15 smithi184 ceph-mon[102844]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:15 smithi196 ceph-mon[76349]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:17 smithi184 ceph-mon[100622]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:17 smithi184 ceph-mon[102844]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:17 smithi196 ceph-mon[76349]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:19.485 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:32:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:19 smithi184 ceph-mon[100622]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:19 smithi184 ceph-mon[102844]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 2h - - 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 16s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 16s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:32:20.017 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 16s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 16s ago 2h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 16s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 16s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 16s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 16s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 16s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 16s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:32:20.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:32:20.019 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 74.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:32:20.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:32:20.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:32:20.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:32:20.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:32:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:19 smithi196 ceph-mon[76349]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:32:20.328 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:32:20.329 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:32:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:32:20.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:32:20.911 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:32:20.911 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:20.911 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:20.911 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:20.911 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:20.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[100622]: from='client.64835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:20.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[100622]: from='client.64841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:20.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1231135547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:20.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[102844]: from='client.64835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:20.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[102844]: from='client.64841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:20.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1231135547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:20.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:20.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:20.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:20.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:20.912 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:20.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:20 smithi196 ceph-mon[76349]: from='client.64835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:20 smithi196 ceph-mon[76349]: from='client.64841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1231135547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[100622]: from='client.64847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[100622]: from='client.54897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[100622]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3405815962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[102844]: from='client.64847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[102844]: from='client.54897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[102844]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3405815962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:21 smithi196 ceph-mon[76349]: from='client.64847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:21 smithi196 ceph-mon[76349]: from='client.54897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:21 smithi196 ceph-mon[76349]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3405815962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:32:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[100622]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[102844]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:23 smithi196 ceph-mon[76349]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[100622]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[102844]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:25 smithi196 ceph-mon[76349]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:26.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:27 smithi184 ceph-mon[100622]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:27 smithi184 ceph-mon[102844]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:27 smithi196 ceph-mon[76349]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:29.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:29 smithi196 ceph-mon[76349]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:29.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:29 smithi184 ceph-mon[100622]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:29 smithi184 ceph-mon[102844]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:31.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:30.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:31.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:30.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:31.029 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:30.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:31.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:31 smithi196 ceph-mon[76349]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:31 smithi184 ceph-mon[100622]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:31 smithi184 ceph-mon[102844]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:32:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:33 smithi184 ceph-mon[100622]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:33 smithi184 ceph-mon[102844]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:33 smithi196 ceph-mon[76349]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:35 smithi184 ceph-mon[100622]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:35 smithi184 ceph-mon[102844]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:35 smithi196 ceph-mon[76349]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:37 smithi184 ceph-mon[102844]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:37 smithi184 ceph-mon[100622]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:37 smithi196 ceph-mon[76349]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:39 smithi184 ceph-mon[100622]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:39 smithi184 ceph-mon[102844]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:39 smithi196 ceph-mon[76349]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:40.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:40.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:40.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:41 smithi184 ceph-mon[102844]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:41 smithi184 ceph-mon[100622]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:41 smithi196 ceph-mon[76349]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:32:43.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:43 smithi196 ceph-mon[76349]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:43.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:43.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:43.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[100622]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[102844]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:45.123 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:45.123 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[102844]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[100622]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:45 smithi196 ceph-mon[76349]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:32:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:47 smithi184 ceph-mon[100622]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:47 smithi184 ceph-mon[102844]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:47 smithi196 ceph-mon[76349]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:49.694 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:49 smithi184 ceph-mon[100622]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:49 smithi184 ceph-mon[102844]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:49 smithi196 ceph-mon[76349]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:51.209 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:50.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:51.209 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:50.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:32:51.209 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:32:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:32:50.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:32:51.209 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 2h - - 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 6s ago 2h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:32:51.760 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 2h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 2h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 6s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 6s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 6s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 6s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 2h 77.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:32:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:32:51.762 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:32:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:51 smithi184 ceph-mon[100622]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:51 smithi184 ceph-mon[102844]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:52.097 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:32:52.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:32:52.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:51 smithi196 ceph-mon[76349]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:32:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:32:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:32:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:32:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:32:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:32:52.370 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:32:52.686 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:32:52.686 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:52.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:52.686 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:52.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:52.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:52 smithi196 ceph-mon[76349]: from='client.64871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:52.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:52 smithi196 ceph-mon[76349]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:52.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:52 smithi196 ceph-mon[76349]: from='client.64883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:52.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2984218907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:52.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/550058779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[100622]: from='client.64871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[100622]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[100622]: from='client.64883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2984218907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/550058779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[102844]: from='client.64871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[102844]: from='client.54915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[102844]: from='client.64883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2984218907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/550058779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:53 smithi184 ceph-mon[102844]: from='client.54933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:53 smithi184 ceph-mon[102844]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:53 smithi184 ceph-mon[100622]: from='client.54933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:53 smithi184 ceph-mon[100622]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:53 smithi196 ceph-mon[76349]: from='client.54933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:53 smithi196 ceph-mon[76349]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:55 smithi184 ceph-mon[100622]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:55 smithi184 ceph-mon[102844]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:55 smithi196 ceph-mon[76349]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:57 smithi184 ceph-mon[100622]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:57 smithi184 ceph-mon[102844]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:57 smithi196 ceph-mon[76349]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:59 smithi184 ceph-mon[100622]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:59 smithi184 ceph-mon[102844]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:59 smithi196 ceph-mon[76349]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:00.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:00.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:00.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:01 smithi184 ceph-mon[102844]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:01 smithi184 ceph-mon[100622]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:01 smithi196 ceph-mon[76349]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:33:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:03 smithi184 ceph-mon[100622]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:03 smithi184 ceph-mon[102844]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:03 smithi196 ceph-mon[76349]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[100622]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[102844]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:05 smithi196 ceph-mon[76349]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:07 smithi196 ceph-mon[76349]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:07 smithi184 ceph-mon[100622]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:07 smithi184 ceph-mon[102844]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:09 smithi196 ceph-mon[76349]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:10.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:09 smithi184 ceph-mon[100622]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:10.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:09 smithi184 ceph-mon[102844]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:11.249 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:10.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:11.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:10.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:11.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:10.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:11 smithi196 ceph-mon[76349]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:11 smithi184 ceph-mon[100622]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:11 smithi184 ceph-mon[102844]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:33:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:13 smithi196 ceph-mon[76349]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:13 smithi184 ceph-mon[100622]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:13 smithi184 ceph-mon[102844]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:14 smithi196 ceph-mon[76349]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[100622]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:14 smithi184 ceph-mon[102844]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:17 smithi184 ceph-mon[100622]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:17 smithi184 ceph-mon[102844]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:17 smithi196 ceph-mon[76349]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:19 smithi184 ceph-mon[100622]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:19 smithi184 ceph-mon[102844]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:19 smithi196 ceph-mon[76349]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:20.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:20.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:20.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:21 smithi184 ceph-mon[100622]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:21 smithi184 ceph-mon[102844]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:21 smithi196 ceph-mon[76349]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:33:22.986 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 2h - - 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 19s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 19s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 19s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 19s ago 2h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 19s ago 2h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:33:23.519 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 19s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 19s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 19s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 19s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 2h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 19s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 19s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:33:23.520 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:33:23.858 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:33:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:33:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:33:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:33:23.859 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:33:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:23 smithi184 ceph-mon[100622]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:23 smithi184 ceph-mon[102844]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:33:24.136 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:33:24.137 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:33:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:23 smithi196 ceph-mon[76349]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:24.447 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:33:24.447 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:24.447 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.64907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.64913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2449445470' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4210249030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.64907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.64913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2449445470' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:24.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4210249030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.64907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.64913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.54957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2449445470' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4210249030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:25 smithi196 ceph-mon[76349]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:25 smithi196 ceph-mon[76349]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[102844]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[102844]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[100622]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[100622]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:27 smithi184 ceph-mon[100622]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:27 smithi184 ceph-mon[102844]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:27 smithi196 ceph-mon[76349]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:29 smithi184 ceph-mon[100622]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:29 smithi184 ceph-mon[102844]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:29 smithi196 ceph-mon[76349]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:31.059 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:30.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:31.059 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:30.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:31.060 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:30.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:31 smithi184 ceph-mon[100622]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:31 smithi184 ceph-mon[102844]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:32.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:31 smithi196 ceph-mon[76349]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:32.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:33:33.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:33 smithi196 ceph-mon[76349]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:33 smithi184 ceph-mon[102844]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:33 smithi184 ceph-mon[100622]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:35.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:35.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:35 smithi184 ceph-mon[100622]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:35 smithi184 ceph-mon[102844]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:35 smithi196 ceph-mon[76349]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:37 smithi184 ceph-mon[100622]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:37 smithi184 ceph-mon[102844]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:37 smithi196 ceph-mon[76349]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:39 smithi184 ceph-mon[100622]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:39 smithi184 ceph-mon[102844]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:39 smithi196 ceph-mon[76349]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:40.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:40.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:40.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:41 smithi184 ceph-mon[102844]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:41 smithi184 ceph-mon[100622]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:41 smithi196 ceph-mon[76349]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:33:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:43 smithi184 ceph-mon[100622]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:43 smithi184 ceph-mon[102844]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:43 smithi196 ceph-mon[76349]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[102844]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[100622]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:45 smithi196 ceph-mon[76349]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:33:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:47 smithi184 ceph-mon[100622]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:47 smithi184 ceph-mon[102844]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:47 smithi196 ceph-mon[76349]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:49 smithi184 ceph-mon[102844]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:49 smithi184 ceph-mon[100622]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:49 smithi196 ceph-mon[76349]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:50.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:50.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:33:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:33:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:33:50.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:33:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:51 smithi184 ceph-mon[100622]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:51 smithi184 ceph-mon[102844]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:52.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:51 smithi196 ceph-mon[76349]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:52.209 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:33:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:33:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:53 smithi184 ceph-mon[100622]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:53 smithi184 ceph-mon[102844]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:53 smithi196 ceph-mon[76349]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:33:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:55.264 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:33:55.264 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T22:33:55.264 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 2h - - 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 11s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 92.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 2h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 2h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 2h 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 2h 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 2h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:33:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 2h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 9s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 9s ago 2h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 2h 78.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 11s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 9s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 11s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:33:55.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 9s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:33:55.574 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:33:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:33:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:33:55.576 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:33:55.845 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:33:55.845 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:33:55.845 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:33:55.846 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:33:55.846 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[100622]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:55.846 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[100622]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:55.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1067991810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:55.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[102844]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:55.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[102844]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:55.847 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1067991810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:55 smithi196 ceph-mon[76349]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:55 smithi196 ceph-mon[76349]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1067991810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:33:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:56.154 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:56.155 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[100622]: from='client.64949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[100622]: from='client.54993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/814459888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[102844]: from='client.64949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[102844]: from='client.54993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/814459888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:56 smithi196 ceph-mon[76349]: from='client.64949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:56 smithi196 ceph-mon[76349]: from='client.54993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/814459888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:57 smithi184 ceph-mon[100622]: from='client.64967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:57 smithi184 ceph-mon[100622]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:57 smithi184 ceph-mon[102844]: from='client.64967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:57 smithi184 ceph-mon[102844]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:57 smithi196 ceph-mon[76349]: from='client.64967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:57 smithi196 ceph-mon[76349]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:59 smithi184 ceph-mon[100622]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:59 smithi184 ceph-mon[102844]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:59 smithi196 ceph-mon[76349]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:00.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:00.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:00.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:02.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:01 smithi184 ceph-mon[100622]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:02.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:01 smithi184 ceph-mon[102844]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:02.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:34:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:01 smithi196 ceph-mon[76349]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:03 smithi184 ceph-mon[100622]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:03 smithi184 ceph-mon[102844]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:03 smithi196 ceph-mon[76349]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:05 smithi184 ceph-mon[100622]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:34:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:05 smithi184 ceph-mon[102844]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:34:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:05 smithi196 ceph-mon[76349]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:34:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:06 smithi196 ceph-mon[76349]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:34:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[100622]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:34:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:07.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:06 smithi184 ceph-mon[102844]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:34:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:09 smithi184 ceph-mon[100622]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:09 smithi184 ceph-mon[102844]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:09 smithi196 ceph-mon[76349]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:10.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:10.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:10.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:11 smithi184 ceph-mon[100622]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T22:34:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:11 smithi184 ceph-mon[102844]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T22:34:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:11 smithi196 ceph-mon[76349]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T22:34:12.279 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:34:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:13 smithi196 ceph-mon[76349]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T22:34:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:13 smithi184 ceph-mon[100622]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T22:34:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:13 smithi184 ceph-mon[102844]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T22:34:14.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:14.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:14.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:14.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:15 smithi184 ceph-mon[100622]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:34:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:15 smithi184 ceph-mon[102844]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:34:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:15 smithi196 ceph-mon[76349]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:34:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:17 smithi184 ceph-mon[100622]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:34:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:17 smithi184 ceph-mon[102844]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:34:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:17 smithi196 ceph-mon[76349]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:34:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:19 smithi184 ceph-mon[100622]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:34:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:19 smithi184 ceph-mon[102844]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:34:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:19 smithi196 ceph-mon[76349]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:34:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:20.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:20.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:20.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:21 smithi184 ceph-mon[100622]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:34:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:21 smithi184 ceph-mon[102844]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:34:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:21 smithi196 ceph-mon[76349]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:34:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:23 smithi184 ceph-mon[100622]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:34:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:23 smithi184 ceph-mon[102844]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:34:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:23 smithi196 ceph-mon[76349]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T22:34:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:25.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:25 smithi196 ceph-mon[76349]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:34:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:25 smithi184 ceph-mon[100622]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:34:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:25 smithi184 ceph-mon[102844]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:34:26.448 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:34:26.971 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:34:26.971 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 2h - - 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 2s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 2s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 2s ago 2h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 2h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 2s ago 2h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 2s ago 2h 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 2s ago 2h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:34:26.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 2s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 2s ago 2h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 2h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 2h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 2h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 2h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 2s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 2s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:34:26.973 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:34:27.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:34:27.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:34:27.282 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:34:27.282 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: from='client.55017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T22:34:27.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[102844]: from='client.64985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: from='client.55017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T22:34:27.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:26 smithi184 ceph-mon[100622]: from='client.64985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: from='client.55017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-20T22:34:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:26 smithi196 ceph-mon[76349]: from='client.64985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:34:27.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:34:27.554 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:34:27.903 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:34:27.903 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:27.903 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[100622]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4235586920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[100622]: from='client.65003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3118498611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[102844]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4235586920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[102844]: from='client.65003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3118498611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:27 smithi196 ceph-mon[76349]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4235586920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:27 smithi196 ceph-mon[76349]: from='client.65003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3118498611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:28 smithi184 ceph-mon[100622]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T22:34:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:28 smithi184 ceph-mon[102844]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T22:34:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:28 smithi196 ceph-mon[76349]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T22:34:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:30.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:30.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:30.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:31 smithi184 ceph-mon[102844]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:31 smithi184 ceph-mon[100622]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:31 smithi196 ceph-mon[76349]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:34:34.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:33 smithi196 ceph-mon[76349]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:34.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:33 smithi184 ceph-mon[100622]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:34.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:33 smithi184 ceph-mon[102844]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:34:34.812 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:34.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:35.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:35.848 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:35 smithi184 ceph-mon[102844]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T22:34:35.849 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:35 smithi184 ceph-mon[100622]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T22:34:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:35 smithi196 ceph-mon[76349]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T22:34:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:37 smithi184 ceph-mon[100622]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-20T22:34:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:37 smithi184 ceph-mon[102844]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-20T22:34:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:37 smithi196 ceph-mon[76349]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-20T22:34:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:39 smithi184 ceph-mon[100622]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:34:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:39 smithi184 ceph-mon[102844]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:34:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:39 smithi196 ceph-mon[76349]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:34:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:40.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:40.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:40.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:41 smithi184 ceph-mon[100622]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:41 smithi184 ceph-mon[102844]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:41 smithi196 ceph-mon[76349]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:34:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:34:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:43 smithi184 ceph-mon[102844]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:34:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:43 smithi184 ceph-mon[100622]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:34:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:43 smithi196 ceph-mon[76349]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:34:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:45.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:45.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:45.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:45 smithi184 ceph-mon[102844]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:34:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:45 smithi184 ceph-mon[100622]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:34:46.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:45 smithi196 ceph-mon[76349]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:34:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:47 smithi196 ceph-mon[76349]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[100622]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:34:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:47 smithi184 ceph-mon[102844]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:49 smithi184 ceph-mon[100622]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:49 smithi184 ceph-mon[102844]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:49 smithi196 ceph-mon[76349]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:50.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:50.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:34:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:34:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:34:50.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:34:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:51 smithi184 ceph-mon[100622]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:51 smithi184 ceph-mon[102844]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:51 smithi196 ceph-mon[76349]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:34:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:34:54.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:53 smithi196 ceph-mon[76349]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:53 smithi184 ceph-mon[100622]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:53 smithi184 ceph-mon[102844]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:54.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:54.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:55.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:55.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:55 smithi184 ceph-mon[100622]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:55.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:55 smithi184 ceph-mon[102844]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:55 smithi196 ceph-mon[76349]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:57 smithi184 ceph-mon[100622]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:34:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:57 smithi184 ceph-mon[102844]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:34:58.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:57 smithi196 ceph-mon[76349]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:34:58.198 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 2h - - 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 12s ago 2h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 12s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:34:58.724 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 2h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 12s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 2h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 12s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 12s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 12s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 12s ago 2h 77.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 12s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 12s ago 2h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:34:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:34:58.726 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 12s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:34:58.726 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:34:58.726 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 12s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:34:59.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:34:59.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:34:59.040 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:34:59.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:34:59.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:34:59.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:34:59.307 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:34:59.308 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:34:59.308 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:34:59.308 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:34:59.619 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:34:59.619 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:59.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:59.619 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:59.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='client.65015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='client.65027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/138558232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1854958690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='client.65015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='client.65027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/138558232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:34:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1854958690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='client.65015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='client.65027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/138558232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:34:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1854958690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:00 smithi184 ceph-mon[100622]: from='client.65039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:00 smithi184 ceph-mon[102844]: from='client.65039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:00.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:00.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:00.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:00 smithi196 ceph-mon[76349]: from='client.65039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:01 smithi184 ceph-mon[100622]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:01 smithi184 ceph-mon[102844]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:01 smithi196 ceph-mon[76349]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:35:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:03 smithi184 ceph-mon[100622]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:03 smithi184 ceph-mon[102844]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:03 smithi196 ceph-mon[76349]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:05 smithi184 ceph-mon[100622]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:05 smithi184 ceph-mon[102844]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:05 smithi196 ceph-mon[76349]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[100622]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:07 smithi184 ceph-mon[102844]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:07.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:07.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:07 smithi196 ceph-mon[76349]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:09 smithi184 ceph-mon[100622]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:09 smithi184 ceph-mon[102844]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:09 smithi196 ceph-mon[76349]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:10.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:10.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:10.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:11 smithi184 ceph-mon[102844]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:11 smithi184 ceph-mon[100622]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:11 smithi196 ceph-mon[76349]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:35:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:13 smithi184 ceph-mon[100622]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:13 smithi184 ceph-mon[102844]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:13 smithi196 ceph-mon[76349]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:15.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:15 smithi196 ceph-mon[76349]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:16.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:15 smithi184 ceph-mon[100622]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:16.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:15 smithi184 ceph-mon[102844]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:17 smithi184 ceph-mon[100622]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:17 smithi184 ceph-mon[102844]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:17 smithi196 ceph-mon[76349]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:19 smithi184 ceph-mon[100622]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:19 smithi184 ceph-mon[102844]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:19 smithi196 ceph-mon[76349]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:20.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:20.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:20.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:21 smithi184 ceph-mon[102844]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:21 smithi184 ceph-mon[100622]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:21 smithi196 ceph-mon[76349]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:35:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:23 smithi184 ceph-mon[100622]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:23 smithi184 ceph-mon[102844]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:23 smithi196 ceph-mon[76349]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:25.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:25 smithi184 ceph-mon[100622]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:25 smithi184 ceph-mon[102844]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:25 smithi196 ceph-mon[76349]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:27 smithi196 ceph-mon[76349]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[100622]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:27 smithi184 ceph-mon[102844]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:29.919 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:35:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:29 smithi184 ceph-mon[100622]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:29 smithi184 ceph-mon[102844]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:29 smithi196 ceph-mon[76349]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:30.448 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:35:30.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 2h - - 2024-09-20T22:35:30.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T22:35:30.448 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 5s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:35:30.448 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 2h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 4s ago 2h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 2h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 5s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 4s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:35:30.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 4s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 4s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 4s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 4s ago 2h 79.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 4s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:35:30.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 4s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:35:30.759 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:35:30.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:35:30.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:35:30.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:35:31.027 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:35:31.028 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:35:31.028 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:35:31.028 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:35:31.028 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:30.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:31.028 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:30.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:31.028 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:30.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:31.339 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:35:31.339 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:31.339 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: from='client.55089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: from='client.65057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3735225372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3723484766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: from='client.55089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: from='client.65057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3735225372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3723484766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: from='client.55089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: from='client.65057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3735225372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3723484766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:35:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:32 smithi184 ceph-mon[100622]: from='client.65075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:32 smithi184 ceph-mon[102844]: from='client.65075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:32 smithi196 ceph-mon[76349]: from='client.65075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:33 smithi184 ceph-mon[100622]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:33 smithi184 ceph-mon[102844]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:33 smithi196 ceph-mon[76349]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:35.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:35.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:35 smithi196 ceph-mon[76349]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:35 smithi184 ceph-mon[100622]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:35 smithi184 ceph-mon[102844]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:37 smithi184 ceph-mon[100622]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:37 smithi184 ceph-mon[102844]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:37 smithi196 ceph-mon[76349]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:39 smithi184 ceph-mon[100622]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:39 smithi184 ceph-mon[102844]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:39 smithi196 ceph-mon[76349]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:40.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:40.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:40.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:41 smithi184 ceph-mon[100622]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:41 smithi184 ceph-mon[102844]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:35:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:41 smithi196 ceph-mon[76349]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:43 smithi184 ceph-mon[100622]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:43 smithi184 ceph-mon[102844]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:43 smithi196 ceph-mon[76349]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:45 smithi196 ceph-mon[76349]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[100622]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[102844]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[102844]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:47 smithi184 ceph-mon[100622]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:35:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:47 smithi196 ceph-mon[76349]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:49 smithi184 ceph-mon[100622]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:49 smithi184 ceph-mon[102844]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:49 smithi196 ceph-mon[76349]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:50.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:50.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:35:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:35:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:35:50.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:35:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:51 smithi184 ceph-mon[100622]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:51 smithi184 ceph-mon[102844]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:51 smithi196 ceph-mon[76349]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:35:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:35:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:53 smithi184 ceph-mon[100622]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:53 smithi184 ceph-mon[102844]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:53.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:53 smithi196 ceph-mon[76349]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:56.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:55 smithi196 ceph-mon[76349]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:56.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:55 smithi184 ceph-mon[100622]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:56.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:55 smithi184 ceph-mon[102844]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:57 smithi184 ceph-mon[100622]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:57 smithi184 ceph-mon[102844]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:57 smithi196 ceph-mon[76349]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:59 smithi184 ceph-mon[100622]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:59 smithi184 ceph-mon[102844]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:59 smithi196 ceph-mon[76349]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:00.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:00.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:00.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:01.635 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:36:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:01 smithi184 ceph-mon[100622]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:01 smithi184 ceph-mon[102844]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:01 smithi196 ceph-mon[76349]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 2h - - 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 15s ago 2h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:36:02.169 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 15s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 2h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 15s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 2h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 15s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 15s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 15s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 15s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 15s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 15s ago 2h 79.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:36:02.170 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:36:02.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 15s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:36:02.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:36:02.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 15s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:36:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:36:02.484 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:36:02.484 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:36:02.484 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:36:02.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:02.484 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:36:02.485 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:36:02.756 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:36:02.756 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:36:02.756 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:36:02.757 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:36:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:02 smithi184 ceph-mon[102844]: from='client.65087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3668902314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:02 smithi184 ceph-mon[100622]: from='client.65087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3668902314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:03.067 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:36:03.067 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:03.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:03.067 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:03.068 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:02 smithi196 ceph-mon[76349]: from='client.65087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3668902314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[100622]: from='client.65093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[100622]: from='client.65099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[100622]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[100622]: from='client.65111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2619951788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[102844]: from='client.65093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[102844]: from='client.65099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[102844]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[102844]: from='client.65111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2619951788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:03 smithi196 ceph-mon[76349]: from='client.65093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:03 smithi196 ceph-mon[76349]: from='client.65099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:03 smithi196 ceph-mon[76349]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:03 smithi196 ceph-mon[76349]: from='client.65111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2619951788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[100622]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[102844]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:05 smithi196 ceph-mon[76349]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:07 smithi196 ceph-mon[76349]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[100622]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:07 smithi184 ceph-mon[102844]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:09 smithi184 ceph-mon[100622]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:09 smithi184 ceph-mon[102844]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:09 smithi196 ceph-mon[76349]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:10.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:10.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:10.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:11 smithi184 ceph-mon[100622]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:11 smithi184 ceph-mon[102844]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:11 smithi196 ceph-mon[76349]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:36:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:13 smithi184 ceph-mon[100622]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:13 smithi184 ceph-mon[102844]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:13 smithi196 ceph-mon[76349]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:15.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:15.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:15 smithi184 ceph-mon[100622]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:15 smithi184 ceph-mon[102844]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:15 smithi196 ceph-mon[76349]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:17.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:17 smithi196 ceph-mon[76349]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:17.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:17 smithi184 ceph-mon[100622]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:17.900 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:17 smithi184 ceph-mon[102844]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:19 smithi184 ceph-mon[100622]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:19 smithi184 ceph-mon[102844]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:19 smithi196 ceph-mon[76349]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:20.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:20.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:20.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:21 smithi184 ceph-mon[100622]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:21 smithi184 ceph-mon[102844]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:22.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:36:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:21 smithi196 ceph-mon[76349]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:23 smithi184 ceph-mon[100622]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:23 smithi184 ceph-mon[102844]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:23 smithi196 ceph-mon[76349]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[100622]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[102844]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:25 smithi196 ceph-mon[76349]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[100622]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[102844]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:27 smithi196 ceph-mon[76349]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:29 smithi184 ceph-mon[102844]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:29 smithi184 ceph-mon[100622]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:30.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:29 smithi196 ceph-mon[76349]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:30.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:30.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:30.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:31 smithi184 ceph-mon[102844]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:31 smithi184 ceph-mon[100622]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:31 smithi196 ceph-mon[76349]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:36:33.364 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 2h - - 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 8s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:36:33.895 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 2h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 7s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 2h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 8s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 7s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 7s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 7s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 7s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 2h 79.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:36:33.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:36:33.897 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 7s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:36:33.897 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:36:33.897 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 7s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:36:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:33 smithi184 ceph-mon[100622]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:33 smithi184 ceph-mon[102844]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:33 smithi196 ceph-mon[76349]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:36:34.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:36:34.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:36:34.207 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:36:34.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:36:34.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:36:34.207 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:36:34.473 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:36:34.474 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:36:34.785 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:36:34.785 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:34.785 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:34.785 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[100622]: from='client.65123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[100622]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2108061534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[102844]: from='client.65123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[102844]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2108061534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:34 smithi196 ceph-mon[76349]: from='client.65123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:34 smithi196 ceph-mon[76349]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2108061534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[100622]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[100622]: from='client.55185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[100622]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1452766355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[102844]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[102844]: from='client.55185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[102844]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1452766355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:35 smithi196 ceph-mon[76349]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:35 smithi196 ceph-mon[76349]: from='client.55185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:35 smithi196 ceph-mon[76349]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1452766355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:37 smithi184 ceph-mon[100622]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:37 smithi184 ceph-mon[102844]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:37 smithi196 ceph-mon[76349]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:39 smithi184 ceph-mon[100622]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:39 smithi184 ceph-mon[102844]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:39 smithi196 ceph-mon[76349]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:40.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:40.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:40.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:41 smithi184 ceph-mon[100622]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:41 smithi184 ceph-mon[102844]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:36:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:41 smithi196 ceph-mon[76349]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:43 smithi184 ceph-mon[100622]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:43 smithi184 ceph-mon[102844]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:43 smithi196 ceph-mon[76349]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[100622]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[102844]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:45 smithi196 ceph-mon[76349]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[100622]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[102844]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:47 smithi196 ceph-mon[76349]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:36:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:49 smithi184 ceph-mon[100622]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:49 smithi184 ceph-mon[102844]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:49 smithi196 ceph-mon[76349]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:50.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:50.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:36:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:36:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:36:50.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:36:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:51 smithi184 ceph-mon[100622]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:36:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:36:52.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:51 smithi184 ceph-mon[102844]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:51 smithi196 ceph-mon[76349]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:53 smithi184 ceph-mon[100622]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:53 smithi184 ceph-mon[102844]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:53 smithi196 ceph-mon[76349]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:55 smithi184 ceph-mon[100622]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:55 smithi184 ceph-mon[102844]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:55 smithi196 ceph-mon[76349]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:58.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:57 smithi196 ceph-mon[76349]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:57 smithi184 ceph-mon[100622]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:57 smithi184 ceph-mon[102844]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:59 smithi184 ceph-mon[100622]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:59.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:59.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:59 smithi184 ceph-mon[102844]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:59.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:59 smithi196 ceph-mon[76349]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:00.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:00.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:00.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:02.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:01 smithi184 ceph-mon[100622]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:02.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:01 smithi184 ceph-mon[102844]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:02.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:37:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:01 smithi196 ceph-mon[76349]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:03 smithi184 ceph-mon[102844]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:04.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:03 smithi184 ceph-mon[100622]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:03 smithi196 ceph-mon[76349]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:05.077 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:05.077 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:05.077 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:37:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:05.612 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:37:05.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 2h - - 2024-09-20T22:37:05.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 2h - - 2024-09-20T22:37:05.612 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 0s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:37:05.612 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 18s ago 2h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 0s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 2h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 2h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 0s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 18s ago 2h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:37:05.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 2h 80.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:37:05.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:37:05.924 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:37:05.924 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:37:05.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:37:05.926 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[102844]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[100622]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:05.926 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:05 smithi196 ceph-mon[76349]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:37:06.195 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:37:06.196 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:37:06.509 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:37:06.509 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:06.509 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:06.509 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:06.509 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:06 smithi196 ceph-mon[76349]: from='client.65159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:06 smithi196 ceph-mon[76349]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:06 smithi196 ceph-mon[76349]: from='client.65171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1296772861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1112998954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[100622]: from='client.65159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[100622]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[100622]: from='client.65171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1296772861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1112998954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[102844]: from='client.65159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[102844]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[102844]: from='client.65171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1296772861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1112998954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:07 smithi196 ceph-mon[76349]: from='client.65183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:07 smithi196 ceph-mon[76349]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[100622]: from='client.65183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[100622]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[102844]: from='client.65183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[102844]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:09 smithi196 ceph-mon[76349]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:09 smithi184 ceph-mon[102844]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:09 smithi184 ceph-mon[100622]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:10.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:10.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:10.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:11 smithi196 ceph-mon[76349]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:11 smithi184 ceph-mon[100622]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:37:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:11 smithi184 ceph-mon[102844]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:13 smithi184 ceph-mon[100622]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:13 smithi184 ceph-mon[102844]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:13 smithi196 ceph-mon[76349]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:15 smithi196 ceph-mon[76349]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:15 smithi184 ceph-mon[100622]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:15 smithi184 ceph-mon[102844]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:17 smithi196 ceph-mon[76349]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:17 smithi184 ceph-mon[100622]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:17 smithi184 ceph-mon[102844]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:20.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:19 smithi196 ceph-mon[76349]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:19 smithi184 ceph-mon[100622]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:19 smithi184 ceph-mon[102844]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:20.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:21.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:20.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:20.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:21 smithi196 ceph-mon[76349]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:21 smithi184 ceph-mon[100622]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:21 smithi184 ceph-mon[102844]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:37:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:23 smithi196 ceph-mon[76349]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:23 smithi184 ceph-mon[100622]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:23 smithi184 ceph-mon[102844]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:25 smithi196 ceph-mon[76349]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[100622]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[102844]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:27.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:26 smithi196 ceph-mon[76349]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:26 smithi184 ceph-mon[100622]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:26 smithi184 ceph-mon[102844]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:29 smithi196 ceph-mon[76349]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:29 smithi184 ceph-mon[100622]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:29 smithi184 ceph-mon[102844]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:30.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:30.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:30.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:31 smithi184 ceph-mon[100622]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:31 smithi184 ceph-mon[102844]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:31 smithi196 ceph-mon[76349]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:37:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:33 smithi184 ceph-mon[100622]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:33 smithi184 ceph-mon[102844]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:33 smithi196 ceph-mon[76349]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:35 smithi184 ceph-mon[100622]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:35 smithi184 ceph-mon[102844]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:35 smithi196 ceph-mon[76349]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:36.807 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 2h - - 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 11s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 2h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 2h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:37:37.331 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 2h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 11s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 10s ago 2h 80.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 11s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:37:37.332 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 11s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:37:37.333 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:37:37.639 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:37:37.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:37:37.639 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:37:37.640 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:37:37.911 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:37:37.912 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:37:37.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[102844]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:37.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[102844]: from='client.65195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:37.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/594292002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:37.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[100622]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:37.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[100622]: from='client.65195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:37.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/594292002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:37 smithi196 ceph-mon[76349]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:37 smithi196 ceph-mon[76349]: from='client.65195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/594292002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:38.219 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:37:38.220 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:38.220 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:38.220 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:38.220 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[102844]: from='client.65201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[102844]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4165047551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[100622]: from='client.65201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[100622]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4165047551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:39.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:38 smithi196 ceph-mon[76349]: from='client.65201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:38 smithi196 ceph-mon[76349]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:39.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4165047551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:39 smithi184 ceph-mon[102844]: from='client.65219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:39 smithi184 ceph-mon[102844]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:39 smithi184 ceph-mon[100622]: from='client.65219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:39 smithi184 ceph-mon[100622]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:39 smithi196 ceph-mon[76349]: from='client.65219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:39 smithi196 ceph-mon[76349]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:40.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:40.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:40.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:40.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:40.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:40.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:41 smithi196 ceph-mon[76349]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:41.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:41 smithi184 ceph-mon[100622]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:41.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:41 smithi184 ceph-mon[102844]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:37:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:43 smithi184 ceph-mon[100622]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:43 smithi184 ceph-mon[102844]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:43 smithi196 ceph-mon[76349]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[102844]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[100622]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:45 smithi196 ceph-mon[76349]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:47 smithi196 ceph-mon[76349]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[102844]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[100622]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:37:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:49 smithi196 ceph-mon[76349]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:49 smithi184 ceph-mon[100622]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:49 smithi184 ceph-mon[102844]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:50.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:50.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:37:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:37:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:37:50.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:51 smithi196 ceph-mon[76349]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:51 smithi184 ceph-mon[100622]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:51 smithi184 ceph-mon[102844]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:37:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:37:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:53 smithi184 ceph-mon[100622]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:53 smithi184 ceph-mon[102844]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:53 smithi196 ceph-mon[76349]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:55 smithi196 ceph-mon[76349]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:55 smithi184 ceph-mon[100622]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:55 smithi184 ceph-mon[102844]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:57 smithi196 ceph-mon[76349]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:57 smithi184 ceph-mon[100622]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:57 smithi184 ceph-mon[102844]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:59 smithi196 ceph-mon[76349]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:59.974 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:00.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:59 smithi184 ceph-mon[100622]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:00.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:00.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:59 smithi184 ceph-mon[102844]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:00.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:00.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:00.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:01.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:00.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:01.935 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:01 smithi184 ceph-mon[100622]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:01.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:01 smithi184 ceph-mon[102844]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:01 smithi196 ceph-mon[76349]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:38:04.126 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:03 smithi184 ceph-mon[100622]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:04.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:03 smithi184 ceph-mon[102844]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:03 smithi196 ceph-mon[76349]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:38:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[100622]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[102844]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:05 smithi196 ceph-mon[76349]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:07 smithi196 ceph-mon[76349]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[100622]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[102844]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:08.516 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 2h - - 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 2h - - 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 3s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 1s ago 2h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 3s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 3s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 2h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:38:09.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 2h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 3s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 1s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 2h 80.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:38:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:38:09.042 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:38:09.353 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:38:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:38:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:38:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:38:09.354 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:38:09.623 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:38:09.624 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:38:09.936 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:38:09.936 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:09.936 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:09.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[100622]: from='client.55269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:09.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[100622]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:09.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[100622]: from='client.65237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:09.936 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4186485432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:09.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[102844]: from='client.55269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:09.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[102844]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:09.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[102844]: from='client.65237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:09.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4186485432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:09 smithi196 ceph-mon[76349]: from='client.55269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:09 smithi196 ceph-mon[76349]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:09 smithi196 ceph-mon[76349]: from='client.65237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4186485432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:10 smithi196 ceph-mon[76349]: from='client.65243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:10 smithi196 ceph-mon[76349]: from='client.65255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3957147771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:10 smithi196 ceph-mon[76349]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[100622]: from='client.65243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[100622]: from='client.65255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3957147771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[100622]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[102844]: from='client.65243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[102844]: from='client.65255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3957147771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-mon[102844]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:10.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:10.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:10.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:38:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:13 smithi184 ceph-mon[100622]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:13 smithi184 ceph-mon[102844]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:14.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:13 smithi196 ceph-mon[76349]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:14.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:38:13.815490094Z level=info msg="Completed cleanup jobs" duration=15.985331ms 2024-09-20T22:38:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:15 smithi184 ceph-mon[100622]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:15 smithi184 ceph-mon[102844]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:15 smithi196 ceph-mon[76349]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:17 smithi184 ceph-mon[100622]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:17 smithi184 ceph-mon[102844]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:17 smithi196 ceph-mon[76349]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:19 smithi196 ceph-mon[76349]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:19 smithi184 ceph-mon[100622]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:19 smithi184 ceph-mon[102844]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:21.081 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:20.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:21.081 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:20.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:21.081 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:20.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:22.071 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:21 smithi184 ceph-mon[100622]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:22.071 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:21 smithi184 ceph-mon[102844]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:22.071 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:38:22.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:21 smithi196 ceph-mon[76349]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:23 smithi196 ceph-mon[76349]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:23.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:23 smithi184 ceph-mon[102844]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:23.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:23 smithi184 ceph-mon[100622]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:25 smithi196 ceph-mon[76349]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:25 smithi184 ceph-mon[100622]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:25 smithi184 ceph-mon[102844]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:27 smithi196 ceph-mon[76349]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[102844]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[100622]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:28 smithi196 ceph-mon[76349]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:28 smithi184 ceph-mon[100622]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:28 smithi184 ceph-mon[102844]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:30 smithi196 ceph-mon[76349]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:30 smithi184 ceph-mon[100622]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:30 smithi184 ceph-mon[102844]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:30.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:30.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:30.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:38:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:33 smithi184 ceph-mon[100622]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:33 smithi184 ceph-mon[102844]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:33 smithi196 ceph-mon[76349]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:35 smithi196 ceph-mon[76349]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:35 smithi184 ceph-mon[100622]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:35 smithi184 ceph-mon[102844]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:37 smithi196 ceph-mon[76349]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:38.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:37 smithi184 ceph-mon[100622]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:38.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:37 smithi184 ceph-mon[102844]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:39 smithi196 ceph-mon[76349]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:40.233 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:39 smithi184 ceph-mon[100622]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:40.233 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:39 smithi184 ceph-mon[102844]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:38:40.760 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:40.760 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 2h - - 2024-09-20T22:38:40.760 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 2h - - 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 13s ago 2h 91.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 13s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 2h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 2h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 14s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 13s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:38:40.761 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 13s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:38:40.762 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:38:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:40.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:40.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:41.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:40.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:41.073 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:38:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:38:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:38:41.074 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:38:41.343 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:38:41.344 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:38:41.344 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:38:41.659 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:38:41.659 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:41.659 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:41.659 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:41.927 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: from='client.55305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.927 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: from='client.65273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.927 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.927 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: from='client.55317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/633951058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1464503070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: from='client.55305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: from='client.65273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: from='client.55317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/633951058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:41.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1464503070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: from='client.55305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: from='client.65273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: from='client.55317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/633951058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:41.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1464503070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:42.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:38:42.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:42 smithi184 ceph-mon[100622]: from='client.65291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:42.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:42 smithi184 ceph-mon[102844]: from='client.65291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:42 smithi196 ceph-mon[76349]: from='client.65291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:44.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:43 smithi184 ceph-mon[100622]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:44.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:43 smithi184 ceph-mon[102844]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:43 smithi196 ceph-mon[76349]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:45 smithi196 ceph-mon[76349]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:45 smithi184 ceph-mon[100622]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:45 smithi184 ceph-mon[102844]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[100622]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:46 smithi184 ceph-mon[102844]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:46 smithi196 ceph-mon[76349]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:38:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:49 smithi184 ceph-mon[100622]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:49 smithi184 ceph-mon[102844]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:49 smithi196 ceph-mon[76349]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:50.829Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:50.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:38:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:38:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:38:50.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:38:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:51 smithi196 ceph-mon[76349]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:51 smithi184 ceph-mon[100622]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:38:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:38:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:51 smithi184 ceph-mon[102844]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:53 smithi196 ceph-mon[76349]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:53 smithi184 ceph-mon[100622]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:53 smithi184 ceph-mon[102844]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:55 smithi196 ceph-mon[76349]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:55 smithi184 ceph-mon[100622]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:55 smithi184 ceph-mon[102844]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:57 smithi196 ceph-mon[76349]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:57 smithi184 ceph-mon[100622]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:57 smithi184 ceph-mon[102844]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:59 smithi196 ceph-mon[76349]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:59 smithi184 ceph-mon[102844]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:59 smithi184 ceph-mon[100622]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:00.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:00.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:00.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:01 smithi196 ceph-mon[76349]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:02.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:01 smithi184 ceph-mon[100622]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:02.170 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:39:02.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:01 smithi184 ceph-mon[102844]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:04.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:03 smithi184 ceph-mon[100622]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:04.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:03 smithi184 ceph-mon[102844]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:03 smithi196 ceph-mon[76349]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:06.123 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:05 smithi184 ceph-mon[100622]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:06.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:05 smithi184 ceph-mon[102844]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:05 smithi196 ceph-mon[76349]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:07 smithi196 ceph-mon[76349]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[100622]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:07 smithi184 ceph-mon[102844]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:09 smithi184 ceph-mon[100622]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:09 smithi184 ceph-mon[102844]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:09 smithi196 ceph-mon[76349]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:10.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:10.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:10.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:11.957 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:39:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:11 smithi184 ceph-mon[100622]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:11 smithi184 ceph-mon[102844]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:12.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:39:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:11 smithi196 ceph-mon[76349]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:12.483 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:12.483 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 2h - - 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 2h - - 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 6s ago 2h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 4s ago 2h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 6s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 4s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 6s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 6s ago 2h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 4s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 6s ago 2h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 6s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 4s ago 2h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:39:12.484 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 4s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 4s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 4s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 4s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 4s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 4s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:39:12.485 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 4s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:39:12.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:39:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:39:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:39:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:39:12.798 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:39:13.073 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:39:13.074 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:39:13.074 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:39:13.074 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:39:13.074 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:39:13.074 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:39:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:39:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:13.384 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:13.384 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: from='client.65303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: from='client.55347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1780908810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/431515089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: from='client.65303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: from='client.55347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1780908810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/431515089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: from='client.65303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: from='client.55347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1780908810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/431515089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[100622]: from='client.55365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[102844]: from='client.55365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:14 smithi196 ceph-mon[76349]: from='client.55365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:15 smithi184 ceph-mon[100622]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:15 smithi184 ceph-mon[102844]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:15 smithi196 ceph-mon[76349]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:17 smithi196 ceph-mon[76349]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:17 smithi184 ceph-mon[100622]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:17 smithi184 ceph-mon[102844]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:19 smithi196 ceph-mon[76349]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:19 smithi184 ceph-mon[100622]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:19 smithi184 ceph-mon[102844]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:20.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:20.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:20.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:22.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:21 smithi196 ceph-mon[76349]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:21 smithi184 ceph-mon[100622]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:21 smithi184 ceph-mon[102844]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:39:23.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:23 smithi184 ceph-mon[100622]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:23.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:23 smithi184 ceph-mon[102844]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:24.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:23 smithi196 ceph-mon[76349]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:25.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:25.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:25 smithi196 ceph-mon[76349]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:26.246 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:25 smithi184 ceph-mon[100622]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:26.247 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:25 smithi184 ceph-mon[102844]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[100622]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:27 smithi184 ceph-mon[102844]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:27 smithi196 ceph-mon[76349]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[100622]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[102844]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:28 smithi196 ceph-mon[76349]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:30.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:30.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:30.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:31 smithi196 ceph-mon[76349]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:31 smithi184 ceph-mon[100622]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:31 smithi184 ceph-mon[102844]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:39:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:33 smithi184 ceph-mon[100622]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:33 smithi184 ceph-mon[102844]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:33 smithi196 ceph-mon[76349]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:35 smithi196 ceph-mon[76349]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:35 smithi184 ceph-mon[100622]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:35 smithi184 ceph-mon[102844]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:37 smithi196 ceph-mon[76349]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:37 smithi184 ceph-mon[100622]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:37 smithi184 ceph-mon[102844]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:39 smithi196 ceph-mon[76349]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:39 smithi184 ceph-mon[100622]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:39 smithi184 ceph-mon[102844]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:40.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:40.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:40.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:41 smithi196 ceph-mon[76349]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:41 smithi184 ceph-mon[100622]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:39:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:41 smithi184 ceph-mon[102844]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:43.675 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:39:43.919 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:43 smithi184 ceph-mon[100622]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:43.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:43 smithi184 ceph-mon[102844]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:44.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:43 smithi196 ceph-mon[76349]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 2h - - 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 2h - - 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 17s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 16s ago 2h 91.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 16s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 2h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:39:44.231 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 16s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 2h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 17s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 16s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 16s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:39:44.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:39:44.233 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:39:44.233 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:39:44.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:39:44.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:39:44.844 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:39:44.845 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:39:45.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[100622]: from='client.65339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:45.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:45.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4261422117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:45.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[102844]: from='client.65339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:45.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:45.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4261422117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:44 smithi196 ceph-mon[76349]: from='client.65339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4261422117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:45.170 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:39:45.170 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:45.170 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:45.170 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:45.170 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[100622]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[100622]: from='client.65351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[100622]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[100622]: from='client.65363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3768813835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[102844]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[102844]: from='client.65351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[102844]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[102844]: from='client.65363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:45.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3768813835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:45 smithi196 ceph-mon[76349]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:45 smithi196 ceph-mon[76349]: from='client.65351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:45 smithi196 ceph-mon[76349]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:45 smithi196 ceph-mon[76349]: from='client.65363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3768813835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:47 smithi196 ceph-mon[76349]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[100622]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:47 smithi184 ceph-mon[102844]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[100622]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:49 smithi184 ceph-mon[102844]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:39:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:49 smithi196 ceph-mon[76349]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:50.834Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:50.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:39:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:39:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:39:50.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:39:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:51 smithi196 ceph-mon[76349]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:51 smithi184 ceph-mon[100622]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:51 smithi184 ceph-mon[102844]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:39:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:39:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:53 smithi184 ceph-mon[100622]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:53 smithi184 ceph-mon[102844]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:53 smithi196 ceph-mon[76349]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:55 smithi196 ceph-mon[76349]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:55 smithi184 ceph-mon[100622]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:55 smithi184 ceph-mon[102844]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:57 smithi196 ceph-mon[76349]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:57 smithi184 ceph-mon[100622]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:57 smithi184 ceph-mon[102844]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:59 smithi196 ceph-mon[76349]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:59 smithi184 ceph-mon[102844]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:59 smithi184 ceph-mon[100622]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:39:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:00.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:00.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:00.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:01 smithi196 ceph-mon[76349]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:01 smithi184 ceph-mon[100622]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:01 smithi184 ceph-mon[102844]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:40:03.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:03 smithi196 ceph-mon[76349]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:03 smithi184 ceph-mon[100622]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:03 smithi184 ceph-mon[102844]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:04.975 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:04.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:05 smithi184 ceph-mon[100622]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:05 smithi184 ceph-mon[102844]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:06.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:05 smithi196 ceph-mon[76349]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[100622]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:07 smithi184 ceph-mon[102844]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:07 smithi196 ceph-mon[76349]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[100622]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:09 smithi184 ceph-mon[102844]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:09 smithi196 ceph-mon[76349]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:10.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:10.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:10.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:11 smithi196 ceph-mon[76349]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:11 smithi184 ceph-mon[100622]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:11 smithi184 ceph-mon[102844]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:40:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:13 smithi184 ceph-mon[100622]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:13 smithi184 ceph-mon[102844]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:13 smithi196 ceph-mon[76349]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:15.471 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 2h - - 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 2h - - 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 9s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 7s ago 2h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:40:15.999 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 9s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 9s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 9s ago 2h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 7s ago 2h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 9s ago 2h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 9s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 7s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 9s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 9s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 9s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 9s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 7s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 7s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:40:16.000 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 7s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 7s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 9s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 7s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 9s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:40:16.001 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 7s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:40:16.103 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:15 smithi184 ceph-mon[100622]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:16.104 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:15 smithi184 ceph-mon[102844]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:16.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:15 smithi196 ceph-mon[76349]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:16.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:40:16.334 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:40:16.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:40:16.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:40:16.603 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:40:16.604 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:40:16.604 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:40:16.604 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:40:16.919 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:16.919 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:16.919 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[100622]: from='client.55413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[100622]: from='client.65381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/392337617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[102844]: from='client.55413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[102844]: from='client.65381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:16.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/392337617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:16 smithi196 ceph-mon[76349]: from='client.55413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:16 smithi196 ceph-mon[76349]: from='client.65381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/392337617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:17 smithi196 ceph-mon[76349]: from='client.65387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:17 smithi196 ceph-mon[76349]: from='client.55437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:17 smithi196 ceph-mon[76349]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1408505846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[100622]: from='client.65387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[100622]: from='client.55437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[100622]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1408505846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[102844]: from='client.65387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[102844]: from='client.55437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[102844]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1408505846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:19 smithi196 ceph-mon[76349]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:19 smithi184 ceph-mon[100622]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:19 smithi184 ceph-mon[102844]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:20.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:20.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:20.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:21 smithi196 ceph-mon[76349]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:21 smithi184 ceph-mon[100622]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:21 smithi184 ceph-mon[102844]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:40:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:23 smithi184 ceph-mon[100622]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:23 smithi184 ceph-mon[102844]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:23 smithi196 ceph-mon[76349]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:25.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:25.136 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:25 smithi184 ceph-mon[100622]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:25 smithi184 ceph-mon[102844]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:26.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:25 smithi196 ceph-mon[76349]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[100622]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:27 smithi184 ceph-mon[102844]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:27 smithi196 ceph-mon[76349]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[100622]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[102844]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:29 smithi196 ceph-mon[76349]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:29.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:30.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:30.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:30.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:31 smithi196 ceph-mon[76349]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:32.201 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:31 smithi184 ceph-mon[100622]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:31 smithi184 ceph-mon[102844]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:40:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:33 smithi184 ceph-mon[100622]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:33 smithi184 ceph-mon[102844]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:33 smithi196 ceph-mon[76349]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:35 smithi196 ceph-mon[76349]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:35 smithi184 ceph-mon[100622]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:35 smithi184 ceph-mon[102844]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:37 smithi196 ceph-mon[76349]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:37 smithi184 ceph-mon[100622]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:37 smithi184 ceph-mon[102844]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:39 smithi196 ceph-mon[76349]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:39 smithi184 ceph-mon[100622]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:39 smithi184 ceph-mon[102844]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:40.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:40.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:40.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:41 smithi196 ceph-mon[76349]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:41 smithi184 ceph-mon[100622]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:41 smithi184 ceph-mon[102844]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:40:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:43 smithi184 ceph-mon[100622]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:43 smithi184 ceph-mon[102844]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:43 smithi196 ceph-mon[76349]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:45.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:45.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:46.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:45 smithi196 ceph-mon[76349]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:46.155 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:45 smithi184 ceph-mon[100622]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:46.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:45 smithi184 ceph-mon[102844]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:40:47.777 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 2h - - 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 2h - - 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 19s ago 2h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 2h 315M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 19s ago 2h 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 2h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 19s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:40:47.778 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:40:47.779 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:40:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[100622]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:47 smithi184 ceph-mon[102844]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:40:48.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:40:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:47 smithi196 ceph-mon[76349]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:40:48.370 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:40:48.370 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:40:48.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:40:48.691 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:48.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:48.692 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:48.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='client.65411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='client.55455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='client.65423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3563970756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2807272654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='client.65411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='client.55455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='client.65423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3563970756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2807272654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='client.65411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='client.55455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='client.65423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3563970756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:40:49.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2807272654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:50.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:49 smithi184 ceph-mon[102844]: from='client.55473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:50.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:49 smithi184 ceph-mon[102844]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:50.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:49 smithi184 ceph-mon[100622]: from='client.55473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:50.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:49 smithi184 ceph-mon[100622]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:49 smithi196 ceph-mon[76349]: from='client.55473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:49 smithi196 ceph-mon[76349]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:50.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:50.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:40:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:40:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:40:50.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:40:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:51 smithi196 ceph-mon[76349]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:51 smithi184 ceph-mon[100622]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:51 smithi184 ceph-mon[102844]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:40:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:40:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:53 smithi184 ceph-mon[100622]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:53 smithi184 ceph-mon[102844]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:53 smithi196 ceph-mon[76349]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:55 smithi196 ceph-mon[76349]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:55 smithi184 ceph-mon[100622]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:55 smithi184 ceph-mon[102844]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:57 smithi196 ceph-mon[76349]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:57 smithi184 ceph-mon[100622]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:57 smithi184 ceph-mon[102844]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:59 smithi196 ceph-mon[76349]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:59 smithi184 ceph-mon[100622]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:59 smithi184 ceph-mon[102844]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:00.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:00.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:00.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:01 smithi196 ceph-mon[76349]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:01 smithi184 ceph-mon[100622]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:02.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:41:02.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:01 smithi184 ceph-mon[102844]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:03 smithi184 ceph-mon[100622]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:03 smithi184 ceph-mon[102844]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:03 smithi196 ceph-mon[76349]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:05 smithi196 ceph-mon[76349]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:05 smithi184 ceph-mon[100622]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:05 smithi184 ceph-mon[102844]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:07 smithi196 ceph-mon[76349]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[100622]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:08.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[102844]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:08.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:08.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[100622]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:09 smithi184 ceph-mon[102844]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:09.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:09 smithi196 ceph-mon[76349]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:10.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:10.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:10.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:11 smithi196 ceph-mon[76349]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:11 smithi184 ceph-mon[100622]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:11 smithi184 ceph-mon[102844]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:41:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:13 smithi196 ceph-mon[76349]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:13 smithi184 ceph-mon[100622]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:13 smithi184 ceph-mon[102844]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:15 smithi196 ceph-mon[76349]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:15 smithi184 ceph-mon[102844]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:15 smithi184 ceph-mon[100622]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:17 smithi196 ceph-mon[76349]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:17 smithi184 ceph-mon[100622]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:17 smithi184 ceph-mon[102844]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:18.987 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 2h - - 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 2h - - 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 12s ago 2h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 10s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:41:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 12s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 2h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 2h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 12s ago 2h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 10s ago 2h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 12s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 12s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 12s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 12s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:41:19.514 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:41:19.515 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 10s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:41:19.515 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 12s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:41:19.515 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:41:19.515 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 12s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:41:19.515 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:41:19.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:41:19.821 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:41:20.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:41:20.094 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:41:20.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:19 smithi184 ceph-mon[100622]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:20.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4238742965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:19 smithi184 ceph-mon[102844]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:20.095 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4238742965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:19 smithi196 ceph-mon[76349]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4238742965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:20.400 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:41:20.400 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:20.400 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:20.400 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:20 smithi196 ceph-mon[76349]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:20 smithi196 ceph-mon[76349]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:20 smithi196 ceph-mon[76349]: from='client.65459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1769762256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[100622]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[100622]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[100622]: from='client.65459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1769762256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[102844]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[102844]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[102844]: from='client.65459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1769762256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:20.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:20.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:20.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:21 smithi196 ceph-mon[76349]: from='client.65471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:21 smithi196 ceph-mon[76349]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:21 smithi184 ceph-mon[100622]: from='client.65471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:21 smithi184 ceph-mon[100622]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:41:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:21 smithi184 ceph-mon[102844]: from='client.65471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:21 smithi184 ceph-mon[102844]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:23 smithi196 ceph-mon[76349]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:23 smithi184 ceph-mon[100622]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:23 smithi184 ceph-mon[102844]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:25 smithi196 ceph-mon[76349]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:25 smithi184 ceph-mon[100622]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:25 smithi184 ceph-mon[102844]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:27.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[102844]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:27.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:27.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:27.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:27.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[100622]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:27.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:27.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:27.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:27 smithi196 ceph-mon[76349]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[100622]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[102844]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:29 smithi196 ceph-mon[76349]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:30.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:30.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:30.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:31 smithi196 ceph-mon[76349]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:31 smithi184 ceph-mon[100622]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:41:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:31 smithi184 ceph-mon[102844]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:33 smithi196 ceph-mon[76349]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:33 smithi184 ceph-mon[100622]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:33 smithi184 ceph-mon[102844]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:35 smithi196 ceph-mon[76349]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:35 smithi184 ceph-mon[100622]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:35 smithi184 ceph-mon[102844]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:37 smithi196 ceph-mon[76349]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:37 smithi184 ceph-mon[100622]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:37 smithi184 ceph-mon[102844]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:39 smithi196 ceph-mon[76349]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:39 smithi184 ceph-mon[100622]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:39 smithi184 ceph-mon[102844]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:40.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:40.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:40.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:41 smithi196 ceph-mon[76349]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:41 smithi184 ceph-mon[100622]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:41 smithi184 ceph-mon[102844]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:41:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:43 smithi196 ceph-mon[76349]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:43 smithi184 ceph-mon[100622]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:43 smithi184 ceph-mon[102844]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:45.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:45.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:45 smithi196 ceph-mon[76349]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:45 smithi184 ceph-mon[100622]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:45 smithi184 ceph-mon[102844]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:47 smithi196 ceph-mon[76349]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[100622]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[102844]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[102844]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:49 smithi184 ceph-mon[100622]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:41:50.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:49 smithi196 ceph-mon[76349]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:50.686 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:41:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:50.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:50.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:41:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:41:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:41:50.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 2h - - 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 2h - - 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 2s ago 2h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 2s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 2h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:41:51.224 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 2h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 2s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 4s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 4s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 4s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 2s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 4s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:41:51.225 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:41:51.536 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:41:51.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:41:51.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:41:51.537 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:41:51.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:41:51.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:41:51.538 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:41:51.807 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:41:51.808 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:41:52.120 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:41:52.120 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:52.120 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:52.120 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:52.120 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:52.120 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:41:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[100622]: from='client.65483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[100622]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2801933902' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[102844]: from='client.65483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[102844]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:52.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2801933902' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:51 smithi196 ceph-mon[76349]: from='client.65483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:51 smithi196 ceph-mon[76349]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:41:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2801933902' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:52 smithi196 ceph-mon[76349]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:52 smithi196 ceph-mon[76349]: from='client.55533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:52 smithi196 ceph-mon[76349]: from='client.55545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/538663035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[100622]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[100622]: from='client.55533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[100622]: from='client.55545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/538663035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[102844]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[102844]: from='client.55533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[102844]: from='client.55545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/538663035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:53 smithi196 ceph-mon[76349]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:53 smithi184 ceph-mon[100622]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:53 smithi184 ceph-mon[102844]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:55 smithi196 ceph-mon[76349]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:55 smithi184 ceph-mon[100622]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:55 smithi184 ceph-mon[102844]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:57 smithi196 ceph-mon[76349]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:57 smithi184 ceph-mon[100622]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:57 smithi184 ceph-mon[102844]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:41:59 smithi196 ceph-mon[76349]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:41:59 smithi184 ceph-mon[100622]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:41:59 smithi184 ceph-mon[102844]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:00.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:00.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:00.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:01 smithi184 ceph-mon[100622]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:42:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:01 smithi184 ceph-mon[102844]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:01 smithi196 ceph-mon[76349]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:03 smithi184 ceph-mon[100622]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:03 smithi184 ceph-mon[102844]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:03 smithi196 ceph-mon[76349]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:04 smithi184 ceph-mon[100622]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:04 smithi184 ceph-mon[102844]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:04 smithi196 ceph-mon[76349]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:07 smithi196 ceph-mon[76349]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[100622]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[102844]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:09 smithi196 ceph-mon[76349]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[100622]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[102844]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:10.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:11.105 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:10.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:11.105 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:10.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:11.105 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:10.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:11 smithi184 ceph-mon[100622]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:11 smithi184 ceph-mon[102844]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:11 smithi196 ceph-mon[76349]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:42:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:13 smithi196 ceph-mon[76349]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:13 smithi184 ceph-mon[100622]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:13 smithi184 ceph-mon[102844]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:15 smithi196 ceph-mon[76349]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:15 smithi184 ceph-mon[100622]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:15 smithi184 ceph-mon[102844]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:17 smithi196 ceph-mon[76349]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:17 smithi184 ceph-mon[100622]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:18.325 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:17 smithi184 ceph-mon[102844]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:19 smithi184 ceph-mon[100622]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:19 smithi184 ceph-mon[102844]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:19 smithi196 ceph-mon[76349]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:21.148 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:20.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:21.148 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:20.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:21.148 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:20.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:21 smithi184 ceph-mon[100622]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:21 smithi184 ceph-mon[102844]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:42:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:21 smithi196 ceph-mon[76349]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:22.418 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:42:22.948 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:42:22.948 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 2h - - 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 2h - - 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 15s ago 2h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 15s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 15s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 15s ago 3h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 2h 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 15s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 15s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 15s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:42:22.949 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 15s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 15s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 15s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 15s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:42:22.950 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:42:23.260 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:42:23.260 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:42:23.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:42:23.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:42:23.261 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:42:23.534 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:42:23.534 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:42:23.534 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:42:23.535 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:42:23.844 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:42:23.844 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:23.844 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:23.844 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:23.844 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[100622]: from='client.55557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[100622]: from='client.55563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[100622]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2162727106' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3022726348' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[102844]: from='client.55557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[102844]: from='client.55563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[102844]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2162727106' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3022726348' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:23 smithi196 ceph-mon[76349]: from='client.55557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:23 smithi196 ceph-mon[76349]: from='client.55563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:23 smithi196 ceph-mon[76349]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2162727106' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3022726348' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:25.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:24 smithi196 ceph-mon[76349]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:24 smithi196 ceph-mon[76349]: from='client.65543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.180 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[100622]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[100622]: from='client.65543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[102844]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[102844]: from='client.65543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:25 smithi184 ceph-mon[100622]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:25 smithi184 ceph-mon[102844]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:25 smithi196 ceph-mon[76349]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:26 smithi184 ceph-mon[100622]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:26 smithi184 ceph-mon[102844]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:26 smithi196 ceph-mon[76349]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[100622]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[102844]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:29 smithi196 ceph-mon[76349]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:29.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:29.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:29.610 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:31.193 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:30.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:31.194 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:30.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:31.194 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:30.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:31 smithi184 ceph-mon[100622]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:31 smithi184 ceph-mon[102844]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:31 smithi196 ceph-mon[76349]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:42:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:33 smithi196 ceph-mon[76349]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:34.179 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:33 smithi184 ceph-mon[100622]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:34.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:33 smithi184 ceph-mon[102844]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:35 smithi196 ceph-mon[76349]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:35 smithi184 ceph-mon[100622]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:35 smithi184 ceph-mon[102844]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:37 smithi184 ceph-mon[100622]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:37 smithi184 ceph-mon[102844]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:37 smithi196 ceph-mon[76349]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:39 smithi184 ceph-mon[100622]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:39 smithi184 ceph-mon[102844]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:39 smithi196 ceph-mon[76349]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:40.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:40.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:40.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:41 smithi184 ceph-mon[100622]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:41 smithi184 ceph-mon[102844]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:42:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:41 smithi196 ceph-mon[76349]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:43 smithi184 ceph-mon[100622]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:43 smithi184 ceph-mon[102844]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:44.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:43 smithi196 ceph-mon[76349]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:44.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:44 smithi184 ceph-mon[102844]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:44 smithi184 ceph-mon[100622]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:45.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:44 smithi196 ceph-mon[76349]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:47 smithi196 ceph-mon[76349]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[102844]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[100622]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:49 smithi196 ceph-mon[76349]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[102844]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[100622]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:42:51.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:50.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:50.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:42:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:42:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:42:50.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:42:52.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:51 smithi184 ceph-mon[100622]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:42:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:42:52.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:51 smithi184 ceph-mon[102844]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:51 smithi196 ceph-mon[76349]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:54.130 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:42:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:53 smithi196 ceph-mon[76349]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:53 smithi184 ceph-mon[100622]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:53 smithi184 ceph-mon[102844]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 2h - - 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 2h - - 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 5s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 5s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:42:54.657 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 3h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 5s ago 2h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 2h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 5s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 7s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 7s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 7s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 5s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 7s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:42:54.658 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:42:54.659 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:42:54.659 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:54.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:42:54.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:54.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:55.241 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:42:55.241 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:42:55.241 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:42:55.241 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:42:55.241 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:42:55.242 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:42:55.553 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:42:55.553 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:55.553 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:55.553 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:55.554 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: from='client.55593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: from='client.65561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: from='client.65567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2606278069' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1580765789' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: from='client.55593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: from='client.65561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: from='client.65567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2606278069' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1580765789' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: from='client.55593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: from='client.65561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: from='client.65567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2606278069' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1580765789' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:56 smithi184 ceph-mon[102844]: from='client.65579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:56 smithi184 ceph-mon[100622]: from='client.65579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:56 smithi196 ceph-mon[76349]: from='client.65579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:57 smithi184 ceph-mon[100622]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:57 smithi184 ceph-mon[102844]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:57 smithi196 ceph-mon[76349]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:58 smithi184 ceph-mon[100622]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:42:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:58 smithi184 ceph-mon[102844]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:42:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:58 smithi196 ceph-mon[76349]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:42:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:00.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:00.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:00.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:01 smithi196 ceph-mon[76349]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:02.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:01 smithi184 ceph-mon[100622]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:02.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:01 smithi184 ceph-mon[102844]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:43:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:03 smithi196 ceph-mon[76349]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:03 smithi184 ceph-mon[100622]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:03 smithi184 ceph-mon[102844]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:05 smithi184 ceph-mon[100622]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:05 smithi184 ceph-mon[102844]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:05 smithi196 ceph-mon[76349]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[100622]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[102844]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:07 smithi196 ceph-mon[76349]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:09.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:08 smithi196 ceph-mon[76349]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:08 smithi184 ceph-mon[100622]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:08 smithi184 ceph-mon[102844]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:11.234 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:10.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:11.235 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:10.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:11.235 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:10.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:11.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:11 smithi184 ceph-mon[100622]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:11.532 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:11 smithi184 ceph-mon[102844]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:11.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:11 smithi196 ceph-mon[76349]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:12.274 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:43:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:13 smithi196 ceph-mon[76349]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:14.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:13 smithi184 ceph-mon[100622]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:13 smithi184 ceph-mon[102844]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:15 smithi184 ceph-mon[100622]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:15 smithi184 ceph-mon[102844]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:15 smithi196 ceph-mon[76349]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:17 smithi184 ceph-mon[100622]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:17 smithi184 ceph-mon[102844]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:17 smithi196 ceph-mon[76349]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:19 smithi184 ceph-mon[100622]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:19 smithi184 ceph-mon[102844]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:19 smithi196 ceph-mon[76349]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:20.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:20.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:20.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:21 smithi184 ceph-mon[100622]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:21 smithi184 ceph-mon[102844]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:43:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:21 smithi196 ceph-mon[76349]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:22 smithi184 ceph-mon[100622]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:22 smithi184 ceph-mon[102844]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:22 smithi196 ceph-mon[76349]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:25.851 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:43:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:25 smithi184 ceph-mon[100622]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:25 smithi184 ceph-mon[102844]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 2h - - 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 18s ago 2h 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 18s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 18s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 18s ago 3h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 2h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 18s ago 2h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:43:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 18s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 18s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 18s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 18s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 17s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 17s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 18s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 17s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 18s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:43:26.381 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 17s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:43:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:25 smithi196 ceph-mon[76349]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:43:26.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:43:26.690 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:43:26.964 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:43:26.965 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:43:26.966 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:43:26.966 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:43:26.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:26 smithi184 ceph-mon[100622]: from='client.55629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:26.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1793702872' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:26.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:26 smithi184 ceph-mon[102844]: from='client.55629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:26.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1793702872' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:27.275 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:43:27.275 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:27.275 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:27.275 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:27.275 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:26 smithi196 ceph-mon[76349]: from='client.55629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1793702872' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='client.65597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='client.65603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1420791530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='client.65597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='client.65603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1420791530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='client.65597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='client.65603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1420791530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:28 smithi184 ceph-mon[100622]: from='client.55653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:28 smithi184 ceph-mon[102844]: from='client.55653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:28 smithi196 ceph-mon[76349]: from='client.55653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[100622]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[102844]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:29 smithi196 ceph-mon[76349]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:31.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:30 smithi196 ceph-mon[76349]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:30 smithi184 ceph-mon[100622]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:30 smithi184 ceph-mon[102844]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:30.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:30.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:30.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:43:34.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:33 smithi184 ceph-mon[100622]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:34.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:33 smithi184 ceph-mon[102844]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:33 smithi196 ceph-mon[76349]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:35 smithi184 ceph-mon[100622]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:35 smithi184 ceph-mon[102844]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:35 smithi196 ceph-mon[76349]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:37 smithi184 ceph-mon[100622]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:37 smithi184 ceph-mon[102844]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:37 smithi196 ceph-mon[76349]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:39 smithi184 ceph-mon[100622]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:39 smithi184 ceph-mon[102844]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:39 smithi196 ceph-mon[76349]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:40 smithi184 ceph-mon[100622]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:40 smithi184 ceph-mon[102844]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:40.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:40.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:40.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:40 smithi196 ceph-mon[76349]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:43:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:43 smithi184 ceph-mon[100622]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:43 smithi184 ceph-mon[102844]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:44.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:43 smithi196 ceph-mon[76349]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:44.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:45 smithi184 ceph-mon[100622]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:45 smithi184 ceph-mon[102844]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:45 smithi196 ceph-mon[76349]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[100622]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[102844]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:47 smithi196 ceph-mon[76349]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:43:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:48 smithi184 ceph-mon[100622]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:48 smithi184 ceph-mon[102844]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:48 smithi196 ceph-mon[76349]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:43:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:50.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:50.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:43:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:43:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:43:50.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:43:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:51 smithi196 ceph-mon[76349]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:51 smithi184 ceph-mon[100622]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:51 smithi184 ceph-mon[102844]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:43:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:43:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:53 smithi184 ceph-mon[100622]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:53 smithi184 ceph-mon[102844]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:53 smithi196 ceph-mon[76349]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:55 smithi184 ceph-mon[100622]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:55 smithi184 ceph-mon[102844]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:55 smithi196 ceph-mon[76349]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:56 smithi184 ceph-mon[100622]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:56 smithi184 ceph-mon[102844]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:56 smithi196 ceph-mon[76349]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:57.571 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:43:58.099 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:58.099 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T22:43:58.099 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T22:43:58.099 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:43:58.099 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 8s ago 2h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 8s ago 2h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 3h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 8s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 8s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 10s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 10s ago 2h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:43:58.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 8s ago 2h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 2h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 8s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:43:58.101 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:43:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:57 smithi184 ceph-mon[102844]: from='client.55665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:57 smithi184 ceph-mon[102844]: from='client.65633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:57 smithi184 ceph-mon[100622]: from='client.55665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:57 smithi184 ceph-mon[100622]: from='client.65633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:57 smithi196 ceph-mon[76349]: from='client.55665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:57 smithi196 ceph-mon[76349]: from='client.65633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:58.412 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:43:58.412 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:43:58.413 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:43:58.682 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:43:58.682 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:43:58.683 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:43:58.994 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:43:58.994 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:58.994 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:58.994 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:58.994 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[100622]: from='client.65639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2564853707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[100622]: from='client.55689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[100622]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[102844]: from='client.65639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2564853707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[102844]: from='client.55689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[102844]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:58 smithi196 ceph-mon[76349]: from='client.65639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2564853707' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:58 smithi196 ceph-mon[76349]: from='client.55689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:58 smithi196 ceph-mon[76349]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:43:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2812359972' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:43:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2812359972' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:43:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2812359972' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:00 smithi184 ceph-mon[100622]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:00 smithi184 ceph-mon[102844]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:00.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:00.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:00.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:00 smithi196 ceph-mon[76349]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:44:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:03 smithi184 ceph-mon[100622]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:03 smithi184 ceph-mon[102844]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:03 smithi196 ceph-mon[76349]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:06.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:06.308 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:05 smithi184 ceph-mon[100622]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:44:06.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:06.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:05 smithi184 ceph-mon[102844]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:44:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:05 smithi196 ceph-mon[76349]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:44:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:06 smithi184 ceph-mon[100622]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:44:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:06 smithi184 ceph-mon[102844]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:44:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:06 smithi196 ceph-mon[76349]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:44:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[102844]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:09 smithi184 ceph-mon[100622]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-20T22:44:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:09 smithi196 ceph-mon[76349]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-20T22:44:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:10.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:10.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:10.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:11.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:11 smithi196 ceph-mon[76349]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T22:44:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:11 smithi184 ceph-mon[100622]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T22:44:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:11 smithi184 ceph-mon[102844]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T22:44:12.301 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:44:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:13 smithi184 ceph-mon[100622]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:44:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:13 smithi184 ceph-mon[102844]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:44:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:14.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:13 smithi196 ceph-mon[76349]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T22:44:14.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:15.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:15.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:14 smithi196 ceph-mon[76349]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:44:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:14 smithi184 ceph-mon[100622]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:44:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:14 smithi184 ceph-mon[102844]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:44:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:17 smithi184 ceph-mon[100622]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:44:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:17 smithi184 ceph-mon[102844]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:44:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:17 smithi196 ceph-mon[76349]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T22:44:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:18 smithi184 ceph-mon[100622]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:44:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:18 smithi184 ceph-mon[102844]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:44:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:18 smithi196 ceph-mon[76349]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T22:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:20 smithi184 ceph-mon[100622]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:20 smithi184 ceph-mon[102844]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:44:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:20.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:20.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:20.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:20 smithi196 ceph-mon[76349]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T22:44:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:44:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:22 smithi184 ceph-mon[100622]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:44:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:22 smithi184 ceph-mon[102844]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:44:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:22 smithi196 ceph-mon[76349]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:44:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:24 smithi184 ceph-mon[100622]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T22:44:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:24 smithi184 ceph-mon[102844]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T22:44:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:24 smithi196 ceph-mon[76349]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T22:44:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:26 smithi184 ceph-mon[100622]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:44:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:26 smithi184 ceph-mon[102844]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:44:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:26 smithi196 ceph-mon[76349]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:44:29.281 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:44:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[100622]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[102844]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:44:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:29 smithi196 ceph-mon[76349]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:29.817 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 3h - - 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 1s ago 2h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 0s ago 2h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 1s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 1s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 1s ago 3h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 1s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 1s ago 2h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 1s ago 3h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:44:29.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 2h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 2h 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 2h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 0s ago 2h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 0s ago 2h 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 2h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 0s ago 2h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 2h 80.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 0s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:44:29.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 0s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:44:30.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:44:30.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:44:30.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:44:30.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:44:30.738 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:44:30.738 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:30.739 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:30.739 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:30.739 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='client.55701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='client.65675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2732681992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2249462130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:30.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:30.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:30.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='client.55701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='client.65675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2732681992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2249462130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='client.55701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='client.65675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2732681992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2249462130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:31 smithi196 ceph-mon[76349]: from='client.55725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:31 smithi196 ceph-mon[76349]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:44:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:31 smithi184 ceph-mon[100622]: from='client.55725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:31 smithi184 ceph-mon[100622]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:44:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:31 smithi184 ceph-mon[102844]: from='client.55725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:31 smithi184 ceph-mon[102844]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:44:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:32 smithi184 ceph-mon[100622]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:44:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:32 smithi184 ceph-mon[102844]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:44:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:32 smithi196 ceph-mon[76349]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:34 smithi184 ceph-mon[100622]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:44:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:34 smithi184 ceph-mon[102844]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:44:35.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:35.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:34 smithi196 ceph-mon[76349]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:44:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:36 smithi184 ceph-mon[100622]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:44:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:36 smithi184 ceph-mon[102844]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:44:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:36 smithi196 ceph-mon[76349]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:44:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:38 smithi184 ceph-mon[100622]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:44:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:38 smithi184 ceph-mon[102844]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:44:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:38 smithi196 ceph-mon[76349]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:40 smithi184 ceph-mon[100622]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:44:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:40 smithi184 ceph-mon[102844]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:44:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:40.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:40.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:40.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:40 smithi196 ceph-mon[76349]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:44:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T22:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:42 smithi184 ceph-mon[100622]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:44:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:42 smithi184 ceph-mon[102844]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:44:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:42 smithi196 ceph-mon[76349]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:44:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:44 smithi196 ceph-mon[76349]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:44 smithi184 ceph-mon[100622]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:44 smithi184 ceph-mon[102844]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:46 smithi184 ceph-mon[100622]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:46 smithi184 ceph-mon[102844]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:47.396 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:46 smithi196 ceph-mon[76349]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[102844]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:49 smithi184 ceph-mon[100622]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:49 smithi196 ceph-mon[76349]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[100622]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-mon[102844]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:50.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:50.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:44:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:44:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:44:50.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:44:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:44:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:50 smithi196 ceph-mon[76349]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:44:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:44:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:44:53.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:52 smithi196 ceph-mon[76349]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:52 smithi184 ceph-mon[100622]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:52 smithi184 ceph-mon[102844]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:55.108 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:55.108 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:54 smithi184 ceph-mon[100622]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:55.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:55.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:54 smithi184 ceph-mon[102844]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:55.332 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:55.332 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:54 smithi196 ceph-mon[76349]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:56 smithi184 ceph-mon[102844]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:56 smithi184 ceph-mon[100622]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:56 smithi196 ceph-mon[76349]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:44:58 smithi184 ceph-mon[100622]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:44:58 smithi184 ceph-mon[102844]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:44:58 smithi196 ceph-mon[76349]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:01.037 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:45:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:00 smithi184 ceph-mon[100622]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:00 smithi184 ceph-mon[102844]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:00.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:00.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:00.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:00 smithi196 ceph-mon[76349]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:01.565 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:01.565 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T22:45:01.565 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 13s ago 2h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 11s ago 2h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 13s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 13s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 13s ago 3h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 11s ago 3h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 13s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 13s ago 2h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 11s ago 2h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 13s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 13s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:45:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 2h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 11s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 11s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 2h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 2h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 11s ago 2h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:45:01.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:01.875 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:45:01.876 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:45:02.145 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:45:02.146 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:45:02.146 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[102844]: from='client.55737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[102844]: from='client.65705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[102844]: from='client.65711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3619142671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[100622]: from='client.55737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[100622]: from='client.65705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[100622]: from='client.65711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3619142671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:01 smithi196 ceph-mon[76349]: from='client.55737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:01 smithi196 ceph-mon[76349]: from='client.65705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:01 smithi196 ceph-mon[76349]: from='client.65711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3619142671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:02.457 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:45:02.457 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:02.457 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:02.457 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:02.457 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[100622]: from='client.65723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1093290246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[100622]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[102844]: from='client.65723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:03.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1093290246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:03.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:02 smithi184 ceph-mon[102844]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:02 smithi196 ceph-mon[76349]: from='client.65723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1093290246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:02 smithi196 ceph-mon[76349]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:45:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:04 smithi184 ceph-mon[100622]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:04 smithi184 ceph-mon[102844]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:04 smithi196 ceph-mon[76349]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:06 smithi184 ceph-mon[100622]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:06 smithi184 ceph-mon[102844]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:06 smithi196 ceph-mon[76349]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[100622]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:09 smithi184 ceph-mon[102844]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:09 smithi196 ceph-mon[76349]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:11.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[100622]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:11.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:11 smithi184 ceph-mon[102844]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:11.230 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:10.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:11.230 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:10.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:11.230 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:10.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:11 smithi196 ceph-mon[76349]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:45:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:12 smithi184 ceph-mon[100622]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:12 smithi184 ceph-mon[102844]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:12 smithi196 ceph-mon[76349]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:14.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:15.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:15.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:14 smithi196 ceph-mon[76349]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:14 smithi184 ceph-mon[100622]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:14 smithi184 ceph-mon[102844]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:16 smithi184 ceph-mon[100622]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:16 smithi184 ceph-mon[102844]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:16 smithi196 ceph-mon[76349]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:18 smithi184 ceph-mon[100622]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:18 smithi184 ceph-mon[102844]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:18 smithi196 ceph-mon[76349]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:20 smithi184 ceph-mon[100622]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:20 smithi184 ceph-mon[102844]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:20.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:20.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:20.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:20 smithi196 ceph-mon[76349]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:45:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:22 smithi184 ceph-mon[100622]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:22 smithi184 ceph-mon[102844]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:22 smithi196 ceph-mon[76349]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:24 smithi184 ceph-mon[102844]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:24 smithi184 ceph-mon[100622]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:24 smithi196 ceph-mon[76349]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:26 smithi184 ceph-mon[102844]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:26 smithi184 ceph-mon[100622]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:26 smithi196 ceph-mon[76349]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[100622]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:29 smithi184 ceph-mon[102844]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:29 smithi196 ceph-mon[76349]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:31.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:30.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:31.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:30.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:31.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:30.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:31.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[102844]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:31.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:31 smithi184 ceph-mon[100622]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:31 smithi196 ceph-mon[76349]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:45:32.756 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:45:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:32 smithi184 ceph-mon[100622]: from='client.65735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:32 smithi184 ceph-mon[100622]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:32 smithi184 ceph-mon[102844]: from='client.65735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:32 smithi184 ceph-mon[102844]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 3h - - 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 4s ago 2h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 3s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 4s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:45:33.318 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 4s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 4s ago 3h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 3h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 4s ago 3h 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 4s ago 2h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 3s ago 2h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 4s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 4s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 4s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 2h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 3s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 3s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 2h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 2h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:45:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 2h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:45:33.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 4s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:45:33.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:45:33.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 4s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:45:33.320 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 3s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:45:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:32 smithi196 ceph-mon[76349]: from='client.65735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:32 smithi196 ceph-mon[76349]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:45:33.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:45:33.656 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:45:33.656 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:45:33.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:45:33.656 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:45:33.923 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:45:33.924 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:45:33.924 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:45:33.924 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:45:33.924 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:45:34.236 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:45:34.236 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:34.237 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:34.237 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:34.237 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[102844]: from='client.65741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[102844]: from='client.65747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/864673364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[100622]: from='client.65741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[100622]: from='client.65747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/864673364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:33 smithi196 ceph-mon[76349]: from='client.65741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:33 smithi196 ceph-mon[76349]: from='client.65747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/864673364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[102844]: from='client.65759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3358762541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[102844]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[100622]: from='client.65759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3358762541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:34 smithi184 ceph-mon[100622]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:34 smithi196 ceph-mon[76349]: from='client.65759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3358762541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:34 smithi196 ceph-mon[76349]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:37.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:36 smithi196 ceph-mon[76349]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:37.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:36 smithi184 ceph-mon[100622]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:37.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:36 smithi184 ceph-mon[102844]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:38 smithi184 ceph-mon[100622]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:38 smithi184 ceph-mon[102844]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:38 smithi196 ceph-mon[76349]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:40 smithi184 ceph-mon[100622]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:40 smithi184 ceph-mon[102844]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:40.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:40.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:40.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:40 smithi196 ceph-mon[76349]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:45:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:42 smithi184 ceph-mon[102844]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:42 smithi184 ceph-mon[100622]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:42 smithi196 ceph-mon[76349]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:44 smithi184 ceph-mon[100622]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:44 smithi184 ceph-mon[102844]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:44 smithi196 ceph-mon[76349]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:46 smithi184 ceph-mon[100622]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:46 smithi184 ceph-mon[102844]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:46 smithi196 ceph-mon[76349]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[100622]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:49 smithi184 ceph-mon[102844]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:49 smithi196 ceph-mon[76349]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:50.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:50.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:45:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:45:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:45:50.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:51 smithi196 ceph-mon[76349]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[100622]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:45:51.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:51 smithi184 ceph-mon[102844]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:45:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T22:45:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:52 smithi184 ceph-mon[100622]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:52 smithi184 ceph-mon[102844]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:52 smithi196 ceph-mon[76349]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:55.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:55.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:54 smithi196 ceph-mon[76349]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:54 smithi184 ceph-mon[100622]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:54 smithi184 ceph-mon[102844]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:57.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:56 smithi196 ceph-mon[76349]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:57.097 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:56 smithi184 ceph-mon[100622]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:57.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:56 smithi184 ceph-mon[102844]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:59.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:59.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:45:59 smithi184 ceph-mon[100622]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:45:59 smithi184 ceph-mon[102844]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:45:59 smithi196 ceph-mon[76349]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:00 smithi184 ceph-mon[100622]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:00 smithi184 ceph-mon[102844]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:00.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:00.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:00.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:00 smithi196 ceph-mon[76349]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:02 smithi184 ceph-mon[100622]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:02 smithi184 ceph-mon[102844]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:02 smithi196 ceph-mon[76349]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:04.534 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 16s ago 2h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 14s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 16s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:46:05.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 16s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 16s ago 3h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 14s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 16s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 16s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 14s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 16s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 16s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 16s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 16s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 2h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 2h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:46:05.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 2h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:46:05.067 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 14s ago 2h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:46:05.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:46:05.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:46:05.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:46:05.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:46:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[100622]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[100622]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[100622]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[102844]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[102844]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:04 smithi184 ceph-mon[102844]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:05.378 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:46:05.378 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:46:05.378 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:46:05.379 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:46:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:04 smithi196 ceph-mon[76349]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:04 smithi196 ceph-mon[76349]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:04 smithi196 ceph-mon[76349]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:46:05.648 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:46:05.649 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:46:05.954 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:46:05.955 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:05.955 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:05.955 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:05.955 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[100622]: from='client.65783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3700146248' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[100622]: from='client.65795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1637676606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[102844]: from='client.65783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3700146248' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[102844]: from='client.65795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1637676606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:05 smithi196 ceph-mon[76349]: from='client.65783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3700146248' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:05 smithi196 ceph-mon[76349]: from='client.65795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1637676606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:06 smithi184 ceph-mon[100622]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:06 smithi184 ceph-mon[102844]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:06 smithi196 ceph-mon[76349]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[100622]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:09 smithi184 ceph-mon[102844]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:09 smithi196 ceph-mon[76349]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:10.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:10.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:10.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[100622]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:11 smithi184 ceph-mon[102844]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:11 smithi196 ceph-mon[76349]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:12 smithi184 ceph-mon[100622]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:12 smithi184 ceph-mon[102844]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:12 smithi196 ceph-mon[76349]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:14 smithi184 ceph-mon[100622]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:14 smithi184 ceph-mon[102844]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:14 smithi196 ceph-mon[76349]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:17.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:16 smithi196 ceph-mon[76349]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:16 smithi184 ceph-mon[100622]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:16 smithi184 ceph-mon[102844]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:18 smithi184 ceph-mon[100622]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:18 smithi184 ceph-mon[102844]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:18 smithi196 ceph-mon[76349]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:20 smithi184 ceph-mon[100622]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:46:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:20 smithi184 ceph-mon[102844]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:46:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:20.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:20.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:20.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:20 smithi196 ceph-mon[76349]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:46:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:46:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:22 smithi184 ceph-mon[100622]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:22 smithi184 ceph-mon[102844]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:22 smithi196 ceph-mon[76349]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:24 smithi184 ceph-mon[100622]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:24 smithi184 ceph-mon[102844]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:24 smithi196 ceph-mon[76349]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:26 smithi184 ceph-mon[100622]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:26 smithi184 ceph-mon[102844]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:26 smithi196 ceph-mon[76349]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:29 smithi184 ceph-mon[100622]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:29 smithi184 ceph-mon[102844]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:29 smithi196 ceph-mon[76349]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:30.864Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:30.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:30.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:31 smithi196 ceph-mon[76349]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[100622]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:31 smithi184 ceph-mon[102844]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:46:33.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:32 smithi184 ceph-mon[100622]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:33.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:32 smithi184 ceph-mon[102844]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:32 smithi196 ceph-mon[76349]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:34 smithi184 ceph-mon[102844]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:34 smithi184 ceph-mon[100622]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:34 smithi196 ceph-mon[76349]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:36.251 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:46:36.778 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:36.778 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T22:46:36.778 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 7s ago 2h 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 6s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 7s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 7s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 7s ago 3h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 7s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 7s ago 2h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 6s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 7s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 7s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 7s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 7s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:46:36.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 6s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 6s ago 2h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 6s ago 2h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 2h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 7s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 7s ago 2h 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:46:36.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:46:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[100622]: from='client.55845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[100622]: from='client.55851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[100622]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[100622]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[102844]: from='client.55845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[102844]: from='client.55851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[102844]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:36 smithi184 ceph-mon[102844]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:37.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:46:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:46:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:46:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:46:37.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:46:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:36 smithi196 ceph-mon[76349]: from='client.55845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:36 smithi196 ceph-mon[76349]: from='client.55851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:36 smithi196 ceph-mon[76349]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:36 smithi196 ceph-mon[76349]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:46:37.674 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:46:37.674 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:37.674 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:37.674 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:37.674 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2024379163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:37 smithi196 ceph-mon[76349]: from='client.65831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/57379461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2024379163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[100622]: from='client.65831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/57379461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2024379163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[102844]: from='client.65831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:38.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/57379461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:39.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:38 smithi184 ceph-mon[100622]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:39.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:38 smithi184 ceph-mon[102844]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:39.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:38 smithi196 ceph-mon[76349]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:40 smithi184 ceph-mon[100622]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:40 smithi184 ceph-mon[102844]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:40.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:40.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:41.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:40.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:40 smithi196 ceph-mon[76349]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:42 smithi184 ceph-mon[100622]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:42 smithi184 ceph-mon[102844]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:42 smithi196 ceph-mon[76349]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:44.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:44.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:44 smithi184 ceph-mon[100622]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:44 smithi184 ceph-mon[102844]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:44 smithi196 ceph-mon[76349]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:46 smithi184 ceph-mon[100622]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:46 smithi184 ceph-mon[102844]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:46 smithi196 ceph-mon[76349]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:48 smithi196 ceph-mon[76349]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:48 smithi184 ceph-mon[100622]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:48 smithi184 ceph-mon[102844]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:50.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:50.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:46:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:46:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:46:50.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[100622]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:51 smithi184 ceph-mon[102844]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:46:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:51 smithi196 ceph-mon[76349]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:46:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:46:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:52 smithi184 ceph-mon[100622]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:52 smithi184 ceph-mon[102844]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:52 smithi196 ceph-mon[76349]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:54 smithi184 ceph-mon[100622]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:54 smithi184 ceph-mon[102844]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:54 smithi196 ceph-mon[76349]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:56 smithi184 ceph-mon[100622]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:56 smithi184 ceph-mon[102844]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:56 smithi196 ceph-mon[76349]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:59.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:59.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:00.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:46:59 smithi184 ceph-mon[100622]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:00.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:46:59 smithi184 ceph-mon[102844]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:46:59 smithi196 ceph-mon[76349]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:01.255 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:00.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:01.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:00.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:01.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:00.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:02.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:01 smithi184 ceph-mon[100622]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:02.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:01 smithi184 ceph-mon[102844]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:02.196 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:47:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:01 smithi196 ceph-mon[76349]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:03 smithi184 ceph-mon[100622]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:03 smithi184 ceph-mon[102844]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:03 smithi196 ceph-mon[76349]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:05 smithi184 ceph-mon[100622]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:05 smithi184 ceph-mon[102844]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:05 smithi196 ceph-mon[76349]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:06 smithi184 ceph-mon[100622]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:06 smithi184 ceph-mon[102844]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:06 smithi196 ceph-mon[76349]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:07.970 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:47:08.492 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:08.492 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 3h - - 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 19s ago 2h 28.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 17s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 19s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 19s ago 3h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 19s ago 3h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 17s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 19s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 19s ago 2h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 17s ago 2h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 19s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 19s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:47:08.493 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 19s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 19s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 17s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 17s ago 2h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 17s ago 2h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 19s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 17s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 19s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:47:08.494 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 17s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:47:08.802 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:47:08.802 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:47:08.802 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:47:08.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:08.802 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:47:08.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:08.804 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:47:08.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:47:08.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:47:08.804 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:47:09.072 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3434610748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:09.073 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3434610748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:47:09.073 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:47:09.074 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:47:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3434610748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:09.388 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:47:09.388 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:09.388 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:09.388 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:09.388 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='client.55887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='client.65855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/226338692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='client.55887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='client.65855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/226338692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='client.55887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='client.65855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:10.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/226338692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:10 smithi196 ceph-mon[76349]: from='client.55905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[100622]: from='client.55905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[102844]: from='client.55905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:10.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:10.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:10.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:11 smithi196 ceph-mon[76349]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:11 smithi184 ceph-mon[100622]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:11 smithi184 ceph-mon[102844]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:47:13.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:12 smithi184 ceph-mon[100622]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:13.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:12 smithi184 ceph-mon[102844]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:12 smithi196 ceph-mon[76349]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:14 smithi184 ceph-mon[100622]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:14 smithi184 ceph-mon[102844]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:15.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:15.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:14 smithi196 ceph-mon[76349]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:16 smithi184 ceph-mon[100622]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:16 smithi184 ceph-mon[102844]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:16 smithi196 ceph-mon[76349]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:19.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:18 smithi196 ceph-mon[76349]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:18 smithi184 ceph-mon[102844]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:19.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:18 smithi184 ceph-mon[100622]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:21.120 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:20 smithi184 ceph-mon[100622]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:21.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:20 smithi184 ceph-mon[102844]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:21.120 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:20.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:21.120 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:20.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:21.121 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:20.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:20 smithi196 ceph-mon[76349]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:47:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:22 smithi184 ceph-mon[100622]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:22 smithi184 ceph-mon[102844]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:22 smithi196 ceph-mon[76349]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:24 smithi184 ceph-mon[100622]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:24 smithi184 ceph-mon[102844]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:24 smithi196 ceph-mon[76349]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:26 smithi184 ceph-mon[100622]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:26 smithi184 ceph-mon[102844]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:26 smithi196 ceph-mon[76349]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:29.358 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[100622]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[102844]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:29 smithi196 ceph-mon[76349]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:30.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:30.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:30.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:47:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[100622]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[102844]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:31 smithi196 ceph-mon[76349]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:33 smithi184 ceph-mon[100622]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:33 smithi184 ceph-mon[102844]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:33 smithi196 ceph-mon[76349]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:34 smithi184 ceph-mon[100622]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:34 smithi184 ceph-mon[102844]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:34 smithi196 ceph-mon[76349]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:36 smithi184 ceph-mon[102844]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:36 smithi184 ceph-mon[100622]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:36 smithi196 ceph-mon[76349]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:38 smithi184 ceph-mon[102844]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:38 smithi184 ceph-mon[100622]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:39.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:38 smithi196 ceph-mon[76349]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:39.681 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:47:40.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:39 smithi196 ceph-mon[76349]: from='client.65879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:39 smithi196 ceph-mon[76349]: from='client.55923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.207 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 10s ago 2h 28.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (2h) 9s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 10s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 10s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 10s ago 3h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 10s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 10s ago 2h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 9s ago 2h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:47:40.208 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 10s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 10s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 10s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 10s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 9s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 9s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 3h 80.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 9s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:47:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 9s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:47:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:39 smithi184 ceph-mon[102844]: from='client.65879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:39 smithi184 ceph-mon[102844]: from='client.55923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:39 smithi184 ceph-mon[100622]: from='client.65879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:39 smithi184 ceph-mon[100622]: from='client.55923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:47:40.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:47:40.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:47:40.529 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:47:40.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:47:40.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:47:40.529 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:47:40.805 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:47:40.805 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:47:40.805 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:47:40.806 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:47:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[102844]: from='client.65891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1776212241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[102844]: from='client.55941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[102844]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[100622]: from='client.65891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1776212241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[100622]: from='client.55941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-mon[100622]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:40.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:40.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:40.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:41.124 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:47:41.124 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:41.124 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:41.124 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:41.124 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:40 smithi196 ceph-mon[76349]: from='client.65891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1776212241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:40 smithi196 ceph-mon[76349]: from='client.55941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:41.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:40 smithi196 ceph-mon[76349]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:42.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2624120519' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:42.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:47:42.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2624120519' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2624120519' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:43 smithi184 ceph-mon[100622]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:43 smithi184 ceph-mon[102844]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:43 smithi196 ceph-mon[76349]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:45 smithi184 ceph-mon[100622]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:45 smithi184 ceph-mon[102844]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:45 smithi196 ceph-mon[76349]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:46 smithi184 ceph-mon[100622]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:46 smithi184 ceph-mon[102844]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:46 smithi196 ceph-mon[76349]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:48 smithi184 ceph-mon[100622]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:48 smithi184 ceph-mon[102844]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:48 smithi196 ceph-mon[76349]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:50.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:50.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:47:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:47:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:47:50.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:47:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[100622]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[102844]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:51 smithi196 ceph-mon[76349]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:47:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:47:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:47:53.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:52 smithi184 ceph-mon[100622]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:53.256 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:52 smithi184 ceph-mon[102844]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:52 smithi196 ceph-mon[76349]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:55.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:55.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:54 smithi196 ceph-mon[76349]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:54 smithi184 ceph-mon[100622]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:54 smithi184 ceph-mon[102844]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:55.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 22:47:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T22:47:55.250Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726861672000 maxt=1726862400000 ulid=01J88SXNJV5VBB9NFGY7F1SYFP duration=22.993971ms 2024-09-20T22:47:55.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 22:47:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T22:47:55.253Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.342767ms 2024-09-20T22:47:55.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 22:47:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T22:47:55.253Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 mint=1726862400000 2024-09-20T22:47:55.641 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 22:47:55 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T22:47:55.258Z caller=head.go:1241 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=4.985184ms 2024-09-20T22:47:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:56 smithi184 ceph-mon[100622]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:56 smithi184 ceph-mon[102844]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:56 smithi196 ceph-mon[76349]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:59.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:00.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:47:59 smithi196 ceph-mon[76349]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:47:59 smithi184 ceph-mon[100622]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:47:59 smithi184 ceph-mon[102844]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:01.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:00 smithi184 ceph-mon[100622]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:01.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:00 smithi184 ceph-mon[102844]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:01.169 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:00.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:01.169 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:00.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:01.169 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:00.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:01.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:00 smithi196 ceph-mon[76349]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:48:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:02 smithi184 ceph-mon[100622]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:02 smithi184 ceph-mon[102844]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:02 smithi196 ceph-mon[76349]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:04 smithi184 ceph-mon[100622]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:04 smithi184 ceph-mon[102844]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:04 smithi196 ceph-mon[76349]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:06 smithi184 ceph-mon[100622]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:06 smithi184 ceph-mon[102844]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:06 smithi196 ceph-mon[76349]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:08 smithi184 ceph-mon[100622]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:08 smithi184 ceph-mon[102844]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:08 smithi196 ceph-mon[76349]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:10.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:10.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:10.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:11.429 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:48:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[100622]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[102844]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:11 smithi196 ceph-mon[76349]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 2s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 2h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 2s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 2s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 2s ago 3h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 0s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:48:11.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 2s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (2h) 2s ago 2h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (2h) 0s ago 2h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 2s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 2s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 2s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 2s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 0s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 0s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 0s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 0s ago 3h 98.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 2s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 0s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:48:11.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 2s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:48:11.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 0s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:48:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:48:12.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:48:12.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:48:12.534 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:48:12.535 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:48:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[102844]: from='client.55953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[102844]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1861291637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[100622]: from='client.55953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[100622]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1861291637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:12.848 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:48:12.848 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:12.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:12.848 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:12.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:12 smithi196 ceph-mon[76349]: from='client.55953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:12 smithi196 ceph-mon[76349]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1861291637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[100622]: from='client.55965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[100622]: from='client.55977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1657036623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[100622]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[102844]: from='client.55965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[102844]: from='client.55977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1657036623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:13 smithi184 ceph-mon[102844]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:13 smithi196 ceph-mon[76349]: from='client.55965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:13 smithi196 ceph-mon[76349]: from='client.55977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1657036623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:13 smithi196 ceph-mon[76349]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:48:13.812693567Z level=info msg="Completed cleanup jobs" duration=12.977816ms 2024-09-20T22:48:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:15 smithi196 ceph-mon[76349]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:15 smithi184 ceph-mon[100622]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:15 smithi184 ceph-mon[102844]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:17 smithi184 ceph-mon[100622]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:17 smithi184 ceph-mon[102844]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:17 smithi196 ceph-mon[76349]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:20 smithi184 ceph-mon[100622]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:20 smithi184 ceph-mon[102844]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:20.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:20 smithi196 ceph-mon[76349]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:20.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:20.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:20.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:22.043 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:22 smithi196 ceph-mon[76349]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:22 smithi184 ceph-mon[100622]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:22.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:22 smithi184 ceph-mon[102844]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:22.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:48:24.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:24 smithi184 ceph-mon[100622]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:24 smithi184 ceph-mon[102844]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:24 smithi196 ceph-mon[76349]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:26 smithi184 ceph-mon[100622]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:26 smithi184 ceph-mon[102844]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:26 smithi196 ceph-mon[76349]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:28 smithi184 ceph-mon[100622]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:28 smithi184 ceph-mon[102844]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:28 smithi196 ceph-mon[76349]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[100622]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[102844]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:30.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:30.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:30.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:30 smithi196 ceph-mon[76349]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:48:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:32 smithi196 ceph-mon[76349]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[100622]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[102844]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:34 smithi184 ceph-mon[100622]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:34 smithi184 ceph-mon[102844]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:34 smithi196 ceph-mon[76349]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:36 smithi196 ceph-mon[76349]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:36 smithi184 ceph-mon[100622]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:36 smithi184 ceph-mon[102844]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:38 smithi196 ceph-mon[76349]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:38 smithi184 ceph-mon[100622]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:38 smithi184 ceph-mon[102844]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:40.503 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:40 smithi196 ceph-mon[76349]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:40 smithi184 ceph-mon[102844]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:40.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:40 smithi184 ceph-mon[100622]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:40.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:40.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:40.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:48:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:42 smithi196 ceph-mon[76349]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:42.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:42 smithi184 ceph-mon[100622]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:42.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:42 smithi184 ceph-mon[102844]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:43.145 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:48:43.684 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:48:43.684 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T22:48:43.684 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 3h - - 2024-09-20T22:48:43.684 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (2h) 14s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:48:43.684 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 12s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 3h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 12s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:48:43.685 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 12s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 12s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 12s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 12s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 12s ago 3h 98.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 12s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:48:43.686 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 12s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:48:44.018 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:48:44.018 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:48:44.019 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:48:44.295 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:48:44.296 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: from='client.71410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: from='client.56001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2524945010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: from='client.71410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: from='client.56001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2524945010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:44.625 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:48:44.625 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:44.625 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:44.625 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:44.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: from='client.71410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: from='client.56001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2524945010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:45 smithi196 ceph-mon[76349]: from='client.56013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1654881226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[100622]: from='client.56013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1654881226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[102844]: from='client.56013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1654881226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:46 smithi196 ceph-mon[76349]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:46 smithi184 ceph-mon[100622]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:46 smithi184 ceph-mon[102844]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:48 smithi196 ceph-mon[76349]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:48 smithi184 ceph-mon[100622]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:48 smithi184 ceph-mon[102844]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[100622]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[102844]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:50.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:50.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:48:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:48:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:48:50.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:50 smithi196 ceph-mon[76349]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:48:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:48:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[100622]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[102844]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:52 smithi196 ceph-mon[76349]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:48:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:54 smithi184 ceph-mon[100622]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:54 smithi184 ceph-mon[102844]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:54 smithi196 ceph-mon[76349]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:56 smithi184 ceph-mon[100622]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:56 smithi184 ceph-mon[102844]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:56 smithi196 ceph-mon[76349]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:58 smithi184 ceph-mon[100622]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:58 smithi184 ceph-mon[102844]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:58 smithi196 ceph-mon[76349]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:00 smithi184 ceph-mon[100622]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:00 smithi184 ceph-mon[102844]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:00.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:00 smithi196 ceph-mon[76349]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:00.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:00.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:00.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:02 smithi184 ceph-mon[100622]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:02 smithi184 ceph-mon[102844]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:02 smithi196 ceph-mon[76349]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:04 smithi184 ceph-mon[100622]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:04 smithi184 ceph-mon[102844]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:04 smithi196 ceph-mon[76349]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:06 smithi184 ceph-mon[102844]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:06 smithi184 ceph-mon[100622]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:06 smithi196 ceph-mon[76349]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:08 smithi184 ceph-mon[100622]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:08 smithi184 ceph-mon[102844]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:08 smithi196 ceph-mon[76349]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[100622]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[102844]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:10.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:10.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:10.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:10 smithi196 ceph-mon[76349]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:49:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:12 smithi196 ceph-mon[76349]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[100622]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[102844]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:14 smithi196 ceph-mon[76349]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:14.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:14 smithi184 ceph-mon[100622]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:14.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:14.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:14 smithi184 ceph-mon[102844]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:14.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:14.913 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 5s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 5s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 5s ago 3h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 3s ago 3h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:49:15.469 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 5s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 5s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 5s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 5s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 5s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 3s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 3s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 3s ago 3h 98.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:49:15.470 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:49:15.471 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 3s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:49:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:49:15.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:15.785 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:49:15.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:49:15.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:49:15.785 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:49:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:15 smithi196 ceph-mon[76349]: from='client.56025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:15 smithi184 ceph-mon[100622]: from='client.56025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:15 smithi184 ceph-mon[102844]: from='client.56025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:16.051 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:49:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:49:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:49:16.052 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:49:16.362 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:49:16.362 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:16.362 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:16.362 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:16.362 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:16 smithi196 ceph-mon[76349]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:16 smithi196 ceph-mon[76349]: from='client.56031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:16 smithi196 ceph-mon[76349]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2215165205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/560011217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[102844]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[102844]: from='client.56031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[102844]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2215165205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/560011217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[100622]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[100622]: from='client.56031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[100622]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2215165205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/560011217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:17 smithi196 ceph-mon[76349]: from='client.56049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:17 smithi184 ceph-mon[100622]: from='client.56049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:17 smithi184 ceph-mon[102844]: from='client.56049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:18 smithi196 ceph-mon[76349]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:18 smithi184 ceph-mon[100622]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:18 smithi184 ceph-mon[102844]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:20 smithi196 ceph-mon[76349]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:20 smithi184 ceph-mon[100622]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:20 smithi184 ceph-mon[102844]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:20.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:20.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:20.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:49:22.674 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:22 smithi196 ceph-mon[76349]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:22 smithi184 ceph-mon[100622]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:22 smithi184 ceph-mon[102844]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:24 smithi184 ceph-mon[100622]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:24 smithi184 ceph-mon[102844]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:24 smithi196 ceph-mon[76349]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:25.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:25.780 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:26.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:26 smithi184 ceph-mon[100622]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:26.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:26 smithi184 ceph-mon[102844]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:26 smithi196 ceph-mon[76349]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:28 smithi184 ceph-mon[100622]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:28 smithi184 ceph-mon[102844]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:28 smithi196 ceph-mon[76349]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[100622]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[102844]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:30.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:31.051 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:30.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:31.051 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:30.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:30 smithi196 ceph-mon[76349]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:49:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[102844]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[100622]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:32 smithi196 ceph-mon[76349]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:34 smithi184 ceph-mon[100622]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:34 smithi184 ceph-mon[102844]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:34 smithi196 ceph-mon[76349]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:36 smithi184 ceph-mon[100622]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:36 smithi184 ceph-mon[102844]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:36 smithi196 ceph-mon[76349]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:38 smithi184 ceph-mon[100622]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:38 smithi184 ceph-mon[102844]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:38 smithi196 ceph-mon[76349]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:40 smithi184 ceph-mon[100622]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:40 smithi184 ceph-mon[102844]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:40.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:40.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:40.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:40 smithi196 ceph-mon[76349]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:49:43.021 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:42 smithi196 ceph-mon[76349]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:42 smithi184 ceph-mon[100622]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:42 smithi184 ceph-mon[102844]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:44 smithi184 ceph-mon[100622]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:44 smithi184 ceph-mon[102844]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:44 smithi196 ceph-mon[76349]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:46.660 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:49:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:46 smithi184 ceph-mon[100622]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:46 smithi184 ceph-mon[102844]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:46 smithi196 ceph-mon[76349]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:47.222 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 3h - - 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 15s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 2h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 15s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 3h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 15s ago 3h 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:49:47.223 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 15s ago 3h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 15s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 15s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 15s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 15s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 15s ago 3h 81.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 15s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 2h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:49:47.224 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 15s ago 2h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:49:47.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:49:47.543 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:49:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[102844]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[102844]: from='client.56067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/988026727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[100622]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[100622]: from='client.56067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/988026727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:47.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:49:47.818 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:49:47.818 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:49:47.818 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:49:47.818 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:49:47.819 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:49:48.133 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:49:48.133 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:48.133 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:48.133 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:48.134 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:47 smithi196 ceph-mon[76349]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:47 smithi196 ceph-mon[76349]: from='client.56067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/988026727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[102844]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[102844]: from='client.66035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[102844]: from='client.56085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2863682273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[100622]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[100622]: from='client.66035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[100622]: from='client.56085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2863682273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:48 smithi196 ceph-mon[76349]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:48 smithi196 ceph-mon[76349]: from='client.66035 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:48 smithi196 ceph-mon[76349]: from='client.56085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2863682273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[102844]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[100622]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:50.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:51.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:50.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:49:51.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:49:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:49:50.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:50 smithi196 ceph-mon[76349]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:49:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:49:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:52 smithi196 ceph-mon[76349]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[100622]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[102844]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:49:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:54 smithi184 ceph-mon[100622]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:54 smithi184 ceph-mon[102844]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:54 smithi196 ceph-mon[76349]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:56 smithi196 ceph-mon[76349]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:56 smithi184 ceph-mon[100622]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:56 smithi184 ceph-mon[102844]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:49:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:58 smithi184 ceph-mon[100622]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:49:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:58 smithi184 ceph-mon[102844]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:49:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:58 smithi196 ceph-mon[76349]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:49:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:49:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:49:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-mon[100622]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-mon[102844]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:00.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:00.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:00.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:00 smithi196 ceph-mon[76349]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:50:03.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:02 smithi196 ceph-mon[76349]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:02 smithi184 ceph-mon[100622]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:02 smithi184 ceph-mon[102844]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:04 smithi196 ceph-mon[76349]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:04 smithi184 ceph-mon[100622]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:04 smithi184 ceph-mon[102844]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:06 smithi184 ceph-mon[100622]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:06 smithi184 ceph-mon[102844]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:06 smithi196 ceph-mon[76349]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:50:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:08 smithi184 ceph-mon[100622]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:08 smithi184 ceph-mon[102844]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:08 smithi196 ceph-mon[76349]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[102844]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:10.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:10.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:10.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[100622]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:10 smithi196 ceph-mon[76349]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T22:50:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:50:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[100622]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[102844]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:12 smithi196 ceph-mon[76349]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:14.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:14.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:14.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:14 smithi196 ceph-mon[76349]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:14 smithi184 ceph-mon[100622]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:14 smithi184 ceph-mon[102844]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:50:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:16 smithi184 ceph-mon[100622]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:16 smithi184 ceph-mon[102844]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:16 smithi196 ceph-mon[76349]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:18.429 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:50:18.958 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 3h - - 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 6s ago 3h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 8s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 8s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 8s ago 3h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 6s ago 3h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 8s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 6s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:50:18.959 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 8s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 8s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 8s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 8s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 6s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 6s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 6s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 6s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 6s ago 3h 81.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:50:18.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:50:19.267 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:50:19.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:50:19.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:50:19.268 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:50:19.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:18 smithi184 ceph-mon[100622]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:19.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:18 smithi184 ceph-mon[102844]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:18 smithi196 ceph-mon[76349]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:50:19.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:50:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:50:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:50:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:50:19.538 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:50:19.847 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:19.847 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:19.847 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:19.847 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:19.847 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[102844]: from='client.66059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[102844]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[102844]: from='client.66071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1869668617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/368119399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[100622]: from='client.66059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[100622]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[100622]: from='client.66071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1869668617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/368119399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:19 smithi196 ceph-mon[76349]: from='client.66059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:19 smithi196 ceph-mon[76349]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:19 smithi196 ceph-mon[76349]: from='client.66071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1869668617' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/368119399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-mon[100622]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-mon[100622]: from='client.56121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-mon[102844]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-mon[102844]: from='client.56121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:20.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:20.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:20.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:20 smithi196 ceph-mon[76349]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:50:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:20 smithi196 ceph-mon[76349]: from='client.56121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:50:23.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:22 smithi196 ceph-mon[76349]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:22 smithi184 ceph-mon[100622]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:22 smithi184 ceph-mon[102844]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:25.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:24 smithi196 ceph-mon[76349]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:24 smithi184 ceph-mon[100622]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:24 smithi184 ceph-mon[102844]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:26 smithi184 ceph-mon[100622]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:26 smithi184 ceph-mon[102844]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:26 smithi196 ceph-mon[76349]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:29.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:28 smithi184 ceph-mon[100622]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:29.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:28 smithi184 ceph-mon[102844]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:28 smithi196 ceph-mon[76349]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[100622]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[102844]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:30.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:31.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:30.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:31.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:30.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:30 smithi196 ceph-mon[76349]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[102844]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[100622]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:32 smithi196 ceph-mon[76349]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:34 smithi184 ceph-mon[102844]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:34 smithi184 ceph-mon[100622]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:34 smithi196 ceph-mon[76349]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:36 smithi184 ceph-mon[100622]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:36 smithi184 ceph-mon[102844]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:36 smithi196 ceph-mon[76349]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:37 smithi184 ceph-mon[102844]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:37 smithi184 ceph-mon[100622]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:37 smithi196 ceph-mon[76349]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:40 smithi196 ceph-mon[76349]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:40 smithi184 ceph-mon[100622]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:40 smithi184 ceph-mon[102844]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:40.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:40.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:40.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:42 smithi184 ceph-mon[100622]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:42 smithi184 ceph-mon[102844]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:50:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:42 smithi196 ceph-mon[76349]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:44.326 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:44 smithi184 ceph-mon[100622]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:44.326 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:44.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:44 smithi184 ceph-mon[102844]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:44.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:44 smithi196 ceph-mon[76349]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:45.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:46 smithi196 ceph-mon[76349]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:46 smithi184 ceph-mon[100622]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:46 smithi184 ceph-mon[102844]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:48 smithi184 ceph-mon[100622]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:48 smithi184 ceph-mon[102844]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:48 smithi196 ceph-mon[76349]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:50.144 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:50:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:50 smithi196 ceph-mon[76349]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:50 smithi184 ceph-mon[100622]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:50 smithi184 ceph-mon[102844]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 3h - - 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 18s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 18s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 0s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 3h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 18s ago 3h 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:50:50.682 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 18s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 18s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 18s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 18s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 18s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 18s ago 3h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 18s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:50:50.683 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:50:50.684 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 18s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:50:50.994 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:50:51.261 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:50:51.261 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:50:51.261 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:50:51.261 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:50:51.261 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:50:51.262 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:50:51.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:50.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:51.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:50.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:50:51.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:50:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:50:50.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:50:51.570 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:51.570 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:51.570 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:51.570 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:51.570 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='client.66095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/388541821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3705533025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='client.66095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/388541821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3705533025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='client.66095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/388541821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:52.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3705533025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:50:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:50:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:52 smithi196 ceph-mon[76349]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:52 smithi196 ceph-mon[76349]: from='client.56157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[100622]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[100622]: from='client.56157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[102844]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[102844]: from='client.56157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:50:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:54 smithi196 ceph-mon[76349]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:54 smithi184 ceph-mon[100622]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:54 smithi184 ceph-mon[102844]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:56 smithi196 ceph-mon[76349]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:57.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:56 smithi184 ceph-mon[100622]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:57.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:56 smithi184 ceph-mon[102844]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:58 smithi196 ceph-mon[76349]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:58 smithi184 ceph-mon[100622]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:58 smithi184 ceph-mon[102844]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:00.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:00 smithi184 ceph-mon[100622]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:00.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:00 smithi184 ceph-mon[102844]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:00 smithi196 ceph-mon[76349]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:00.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:00.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:00.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:51:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:02 smithi196 ceph-mon[76349]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:02 smithi184 ceph-mon[100622]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:02 smithi184 ceph-mon[102844]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:04 smithi196 ceph-mon[76349]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:04 smithi184 ceph-mon[100622]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:04 smithi184 ceph-mon[102844]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:06 smithi184 ceph-mon[100622]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:06 smithi184 ceph-mon[102844]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:06 smithi196 ceph-mon[76349]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:08.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:08 smithi184 ceph-mon[100622]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:08.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:08 smithi184 ceph-mon[102844]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:08 smithi196 ceph-mon[76349]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:10 smithi184 ceph-mon[100622]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:10 smithi184 ceph-mon[102844]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:10.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:10.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:10.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:10 smithi196 ceph-mon[76349]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:51:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[100622]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[102844]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:12 smithi196 ceph-mon[76349]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:14 smithi184 ceph-mon[100622]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:14 smithi184 ceph-mon[102844]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:14 smithi196 ceph-mon[76349]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:16.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:16.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:16 smithi184 ceph-mon[100622]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:16 smithi184 ceph-mon[102844]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:17.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:16 smithi196 ceph-mon[76349]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:18 smithi184 ceph-mon[100622]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:18 smithi184 ceph-mon[102844]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:18 smithi196 ceph-mon[76349]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:20 smithi184 ceph-mon[100622]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:20 smithi184 ceph-mon[102844]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:20.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:20.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:21.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:20.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:20 smithi196 ceph-mon[76349]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:21.868 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:51:22.395 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:51:22.395 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 3h - - 2024-09-20T22:51:22.395 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T22:51:22.395 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 11s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 11s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 11s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 11s ago 3h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 9s ago 3h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 11s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 11s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 11s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 11s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 11s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 11s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 9s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:51:22.396 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 9s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 9s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 9s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 9s ago 3h 76.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 9s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 11s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:51:22.397 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 9s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:51:22.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:22.705 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:51:22.706 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:51:22.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:51:22.706 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:51:22.975 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:51:22.975 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:51:22.976 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[102844]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[102844]: from='client.56169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3743624567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[100622]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[100622]: from='client.56169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:22.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3743624567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:22 smithi196 ceph-mon[76349]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:22 smithi196 ceph-mon[76349]: from='client.56169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3743624567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:23.283 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:51:23.283 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:23.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:23.283 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:23.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[100622]: from='client.66137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[100622]: from='client.56181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[100622]: from='client.66155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3049378778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[102844]: from='client.66137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[102844]: from='client.56181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[102844]: from='client.66155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3049378778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:23 smithi196 ceph-mon[76349]: from='client.66137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:23 smithi196 ceph-mon[76349]: from='client.56181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:23 smithi196 ceph-mon[76349]: from='client.66155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3049378778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:25.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:24 smithi196 ceph-mon[76349]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:24 smithi184 ceph-mon[100622]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:24 smithi184 ceph-mon[102844]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:26.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:26.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:26 smithi196 ceph-mon[76349]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:26 smithi184 ceph-mon[102844]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:26 smithi184 ceph-mon[100622]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:28 smithi184 ceph-mon[100622]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:28 smithi184 ceph-mon[102844]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:28 smithi196 ceph-mon[76349]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:30.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:30 smithi184 ceph-mon[100622]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:30.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:30 smithi184 ceph-mon[102844]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:30 smithi196 ceph-mon[76349]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:30.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:30.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:30.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:32.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:32 smithi196 ceph-mon[76349]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.220 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[100622]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:33.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.221 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[102844]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:33.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:33.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:34 smithi196 ceph-mon[76349]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:34 smithi184 ceph-mon[100622]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:34 smithi184 ceph-mon[102844]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:36 smithi196 ceph-mon[76349]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:36 smithi184 ceph-mon[100622]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:36 smithi184 ceph-mon[102844]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:38 smithi196 ceph-mon[76349]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:38 smithi184 ceph-mon[100622]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:38 smithi184 ceph-mon[102844]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:40 smithi196 ceph-mon[76349]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:40 smithi184 ceph-mon[100622]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:40 smithi184 ceph-mon[102844]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:40.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:40.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:40.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:51:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:42 smithi196 ceph-mon[76349]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:42 smithi184 ceph-mon[100622]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:42 smithi184 ceph-mon[102844]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:44.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:45.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:44 smithi196 ceph-mon[76349]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:44 smithi184 ceph-mon[100622]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:44 smithi184 ceph-mon[102844]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:46 smithi196 ceph-mon[76349]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:46 smithi184 ceph-mon[102844]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:46 smithi184 ceph-mon[100622]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:49.250 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:48 smithi184 ceph-mon[100622]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:49.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:48 smithi184 ceph-mon[102844]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:48 smithi196 ceph-mon[76349]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:51.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:50 smithi184 ceph-mon[100622]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:51.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:50 smithi184 ceph-mon[102844]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:51.238 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:50.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:51.238 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:50.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:51:51.238 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:51:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:51:50.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:51:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:50 smithi196 ceph-mon[76349]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.235 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:51:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:51:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[100622]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[102844]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:52 smithi196 ceph-mon[76349]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:51:53.569 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 3h - - 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 1s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 3s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 1s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 3s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:51:54.098 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 3h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 1s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 1s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 1s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 1s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 1s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 1s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 1s ago 3h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:51:54.099 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 1s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:51:54.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:51:54.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 1s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:51:54.407 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:51:54.407 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:51:54.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:51:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:51:54.409 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:51:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:51:54.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:51:54.987 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:51:54.987 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:54.988 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:54.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:54 smithi196 ceph-mon[76349]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:54 smithi196 ceph-mon[76349]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:54 smithi196 ceph-mon[76349]: from='client.66173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3430496494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[100622]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[100622]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[100622]: from='client.66173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3430496494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[102844]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[102844]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[102844]: from='client.66173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3430496494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[100622]: from='client.56217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[100622]: from='client.66191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4024883404' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[102844]: from='client.56217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[102844]: from='client.66191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4024883404' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:55 smithi196 ceph-mon[76349]: from='client.56217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:55 smithi196 ceph-mon[76349]: from='client.66191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4024883404' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:56 smithi184 ceph-mon[100622]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:56 smithi184 ceph-mon[102844]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:56 smithi196 ceph-mon[76349]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:57 smithi184 ceph-mon[100622]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:57 smithi184 ceph-mon[102844]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:57 smithi196 ceph-mon[76349]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:51:59 smithi184 ceph-mon[100622]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:51:59 smithi184 ceph-mon[102844]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:51:59 smithi196 ceph-mon[76349]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:00.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:00.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:00.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:02 smithi184 ceph-mon[100622]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:02 smithi184 ceph-mon[102844]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:52:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:02 smithi196 ceph-mon[76349]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:04 smithi196 ceph-mon[76349]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:04.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:04 smithi184 ceph-mon[100622]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:04.390 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:04 smithi184 ceph-mon[102844]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:06.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:06 smithi196 ceph-mon[76349]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:06 smithi184 ceph-mon[100622]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:06 smithi184 ceph-mon[102844]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:08.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:08 smithi196 ceph-mon[76349]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:08 smithi184 ceph-mon[100622]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:08 smithi184 ceph-mon[102844]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:10 smithi184 ceph-mon[102844]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:10 smithi184 ceph-mon[100622]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:10 smithi196 ceph-mon[76349]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:10.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:10.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:10.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:12.100 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T22:52:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:12 smithi196 ceph-mon[76349]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[100622]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[102844]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:14 smithi184 ceph-mon[100622]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:14 smithi184 ceph-mon[102844]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:14 smithi196 ceph-mon[76349]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:16 smithi184 ceph-mon[100622]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:16 smithi184 ceph-mon[102844]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:16 smithi196 ceph-mon[76349]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:18 smithi196 ceph-mon[76349]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:18 smithi184 ceph-mon[100622]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:18 smithi184 ceph-mon[102844]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:20 smithi196 ceph-mon[76349]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:20 smithi184 ceph-mon[100622]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:20 smithi184 ceph-mon[102844]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:20.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:20.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:20.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:52:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:22 smithi196 ceph-mon[76349]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:22 smithi184 ceph-mon[100622]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:22 smithi184 ceph-mon[102844]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:24 smithi196 ceph-mon[76349]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:25.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:24 smithi184 ceph-mon[100622]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:25.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:24 smithi184 ceph-mon[102844]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:25.288 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 14s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:52:25.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 14s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 14s ago 3h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 14s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 14s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 14s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 14s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 14s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:52:25.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:52:25.819 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 13s ago 3h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:52:25.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:52:25.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:52:25.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:52:25.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:52:26.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:52:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:52:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:52:26.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:52:26.130 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:26.130 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:26.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:52:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:52:26.396 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:52:26.707 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:52:26.707 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:26.707 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:26.708 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:26.708 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: from='client.56247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: from='client.66215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1513669550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3048449632' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: from='client.56247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: from='client.66215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1513669550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3048449632' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: from='client.56247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: from='client.66215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1513669550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:27.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3048449632' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:28.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:27 smithi196 ceph-mon[76349]: from='client.66227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:27 smithi184 ceph-mon[100622]: from='client.66227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:27 smithi184 ceph-mon[102844]: from='client.66227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:28 smithi196 ceph-mon[76349]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:28 smithi184 ceph-mon[100622]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:28 smithi184 ceph-mon[102844]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:30.104 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:30.104 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:30 smithi196 ceph-mon[76349]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.145 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:30 smithi184 ceph-mon[100622]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.146 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:30 smithi184 ceph-mon[102844]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.146 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:30.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:31.146 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:30.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:31.146 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:30.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:32.244 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:52:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[100622]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[102844]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:32 smithi196 ceph-mon[76349]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:34 smithi184 ceph-mon[100622]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:34 smithi184 ceph-mon[102844]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:34 smithi196 ceph-mon[76349]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:36 smithi184 ceph-mon[100622]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:36 smithi184 ceph-mon[102844]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:36 smithi196 ceph-mon[76349]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:38 smithi184 ceph-mon[100622]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:38 smithi184 ceph-mon[102844]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:38 smithi196 ceph-mon[76349]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:40 smithi184 ceph-mon[100622]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:40 smithi184 ceph-mon[102844]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:40.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:40.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:40.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:40 smithi196 ceph-mon[76349]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:52:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:42 smithi184 ceph-mon[100622]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:42 smithi184 ceph-mon[102844]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:42 smithi196 ceph-mon[76349]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:44 smithi184 ceph-mon[100622]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:44 smithi184 ceph-mon[102844]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:44 smithi196 ceph-mon[76349]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:47.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:46 smithi196 ceph-mon[76349]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:46 smithi184 ceph-mon[100622]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:46 smithi184 ceph-mon[102844]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:48 smithi184 ceph-mon[100622]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:48 smithi184 ceph-mon[102844]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:48 smithi196 ceph-mon[76349]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:51.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:50 smithi184 ceph-mon[100622]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:51.135 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:50 smithi184 ceph-mon[102844]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:51.135 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:50.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:51.135 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:50.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:52:51.135 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:52:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:52:50.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:52:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:50 smithi196 ceph-mon[76349]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:52.136 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:52:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:52:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[100622]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[102844]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:52 smithi196 ceph-mon[76349]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:52:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[100622]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[102844]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:54 smithi196 ceph-mon[76349]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:54.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:52:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:56 smithi184 ceph-mon[100622]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:56 smithi184 ceph-mon[102844]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:56 smithi196 ceph-mon[76349]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:57.004 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:52:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:57 smithi184 ceph-mon[102844]: from='client.66239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:57 smithi184 ceph-mon[100622]: from='client.66239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:57 smithi196 ceph-mon[76349]: from='client.66239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:57.589 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:52:57.589 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 3h - - 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 6s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 6s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 6s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 6s ago 3h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 4s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 6s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 6s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 4s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 4s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 4s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 4s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 4s ago 3h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 4s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:52:57.591 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 4s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:52:57.900 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:52:57.900 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:52:57.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:52:57.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:52:57.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:52:57.902 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:52:58.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:52:58.171 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[100622]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[100622]: from='client.56283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[100622]: from='client.66251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/293054150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[102844]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[102844]: from='client.56283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[102844]: from='client.66251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/293054150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:58 smithi196 ceph-mon[76349]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:58 smithi196 ceph-mon[76349]: from='client.56283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:58 smithi196 ceph-mon[76349]: from='client.66251 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/293054150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:59 smithi196 ceph-mon[76349]: from='client.66263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3634871949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[100622]: from='client.66263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3634871949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[102844]: from='client.66263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3634871949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:00 smithi196 ceph-mon[76349]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:00 smithi184 ceph-mon[100622]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:00 smithi184 ceph-mon[102844]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:00.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:00.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:00.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:02 smithi184 ceph-mon[100622]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:02 smithi184 ceph-mon[102844]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:53:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:02 smithi196 ceph-mon[76349]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:04 smithi196 ceph-mon[76349]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:04.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:04 smithi184 ceph-mon[100622]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:04.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:04 smithi184 ceph-mon[102844]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:06 smithi196 ceph-mon[76349]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:06 smithi184 ceph-mon[100622]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:06 smithi184 ceph-mon[102844]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:08.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:08 smithi196 ceph-mon[76349]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:08 smithi184 ceph-mon[100622]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:08 smithi184 ceph-mon[102844]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:10.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:10 smithi196 ceph-mon[76349]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:10 smithi184 ceph-mon[100622]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:10 smithi184 ceph-mon[102844]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:10.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:10.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:10.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:12.034 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:53:12.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[100622]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:12.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[102844]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:12.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:12 smithi196 ceph-mon[76349]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[100622]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[102844]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:14.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:14 smithi196 ceph-mon[76349]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:14.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:14.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:16 smithi184 ceph-mon[100622]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:16 smithi184 ceph-mon[102844]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:16 smithi196 ceph-mon[76349]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:18 smithi184 ceph-mon[100622]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:18 smithi184 ceph-mon[102844]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:18 smithi196 ceph-mon[76349]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:20 smithi184 ceph-mon[100622]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:20 smithi184 ceph-mon[102844]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:20 smithi196 ceph-mon[76349]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:20.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:20.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:20.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:22 smithi184 ceph-mon[102844]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:22 smithi184 ceph-mon[100622]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:53:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:22 smithi196 ceph-mon[76349]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:24.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:24 smithi184 ceph-mon[100622]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:24.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:24 smithi184 ceph-mon[102844]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:24 smithi196 ceph-mon[76349]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:26 smithi184 ceph-mon[100622]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:26 smithi184 ceph-mon[102844]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:26 smithi196 ceph-mon[76349]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:28.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:28 smithi196 ceph-mon[76349]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:28 smithi184 ceph-mon[100622]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:28 smithi184 ceph-mon[102844]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:28.774 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 17s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:53:29.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 17s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 17s ago 3h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 16s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 17s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 17s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 17s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 17s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 17s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:53:29.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:53:29.308 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 16s ago 3h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:53:29.308 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:53:29.308 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:53:29.308 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 17s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:53:29.308 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:53:29.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:29 smithi196 ceph-mon[76349]: from='client.66275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:29.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:29 smithi184 ceph-mon[100622]: from='client.66275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:29 smithi184 ceph-mon[102844]: from='client.66275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:53:29.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:53:29.617 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:53:29.885 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:53:29.886 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:53:30.196 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:53:30.196 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:30.196 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: from='client.66281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/436066821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: from='client.66299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1651242956' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: from='client.66281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/436066821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: from='client.66299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1651242956' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: from='client.66281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/436066821' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: from='client.66299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1651242956' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:30.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:30.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:30.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:53:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:32 smithi196 ceph-mon[76349]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[100622]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:32.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[102844]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:32.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:32.968 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[100622]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[102844]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:34 smithi196 ceph-mon[76349]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:36 smithi196 ceph-mon[76349]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:36 smithi184 ceph-mon[100622]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:36 smithi184 ceph-mon[102844]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:38 smithi196 ceph-mon[76349]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:38 smithi184 ceph-mon[100622]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:38 smithi184 ceph-mon[102844]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:40 smithi196 ceph-mon[76349]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:40 smithi184 ceph-mon[100622]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:40 smithi184 ceph-mon[102844]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:40.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:40.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:40.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:53:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:42 smithi196 ceph-mon[76349]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:42 smithi184 ceph-mon[100622]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:42 smithi184 ceph-mon[102844]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:44 smithi196 ceph-mon[76349]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:44 smithi184 ceph-mon[100622]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:44 smithi184 ceph-mon[102844]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:46.323 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:46 smithi184 ceph-mon[100622]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:46 smithi196 ceph-mon[76349]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:46 smithi184 ceph-mon[102844]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:48 smithi196 ceph-mon[76349]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:48 smithi184 ceph-mon[100622]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:48 smithi184 ceph-mon[102844]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:50.576 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:50 smithi196 ceph-mon[76349]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:50 smithi184 ceph-mon[100622]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:50 smithi184 ceph-mon[102844]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:50.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:50.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:53:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:53:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:53:50.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:53:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:53:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:53:52.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[102844]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:52.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:52.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:52.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:52.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[100622]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:52.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:52.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:52.869 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:52 smithi196 ceph-mon[76349]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[100622]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:54.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[102844]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:54.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:54 smithi196 ceph-mon[76349]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:53:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:56 smithi184 ceph-mon[100622]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:56 smithi184 ceph-mon[102844]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:56 smithi196 ceph-mon[76349]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:58 smithi184 ceph-mon[100622]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:58 smithi184 ceph-mon[102844]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:58 smithi196 ceph-mon[76349]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:00.496 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:54:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:00 smithi184 ceph-mon[100622]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:00 smithi184 ceph-mon[102844]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:00 smithi196 ceph-mon[76349]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:01.023 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:54:01.023 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T22:54:01.023 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T22:54:01.023 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:54:01.023 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 9s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (2h) 9s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 9s ago 3h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 7s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 9s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 9s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 9s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 9s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 9s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 7s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 7s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:54:01.024 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 7s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 7s ago 3h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 7s ago 3h 79.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 7s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:54:01.025 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 7s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:54:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:00.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:00.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:00.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:54:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:54:01.336 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:54:01.608 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:54:01.609 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:54:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[100622]: from='client.56349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[100622]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/292900154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[102844]: from='client.56349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[102844]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/292900154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:01 smithi196 ceph-mon[76349]: from='client.56349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:01 smithi196 ceph-mon[76349]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/292900154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:01.914 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:54:01.914 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:01.914 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:01.914 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:01.915 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:54:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[100622]: from='client.66323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[100622]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[100622]: from='client.66335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3156791490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[102844]: from='client.66323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[102844]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[102844]: from='client.66335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3156791490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:02 smithi196 ceph-mon[76349]: from='client.66323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:02 smithi196 ceph-mon[76349]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:02 smithi196 ceph-mon[76349]: from='client.66335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3156791490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:04 smithi184 ceph-mon[100622]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:04 smithi184 ceph-mon[102844]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:04 smithi196 ceph-mon[76349]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:06 smithi184 ceph-mon[100622]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:54:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:06 smithi184 ceph-mon[102844]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:54:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:06 smithi196 ceph-mon[76349]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T22:54:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:08 smithi184 ceph-mon[100622]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:54:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:08 smithi184 ceph-mon[102844]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:54:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:08 smithi196 ceph-mon[76349]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T22:54:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:10 smithi184 ceph-mon[100622]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T22:54:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:10 smithi184 ceph-mon[102844]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T22:54:10.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:10 smithi196 ceph-mon[76349]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T22:54:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:10.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:10.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:10.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:54:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[100622]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T22:54:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[102844]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:12 smithi196 ceph-mon[76349]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T22:54:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[100622]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[102844]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:14 smithi196 ceph-mon[76349]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T22:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:15.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:15.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:16 smithi196 ceph-mon[76349]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:54:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:16 smithi184 ceph-mon[100622]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:54:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:16 smithi184 ceph-mon[102844]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T22:54:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:18 smithi196 ceph-mon[76349]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T22:54:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:18 smithi184 ceph-mon[100622]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T22:54:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:18 smithi184 ceph-mon[102844]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T22:54:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:20 smithi196 ceph-mon[76349]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:54:20.902 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:20 smithi184 ceph-mon[100622]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:54:20.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:20 smithi184 ceph-mon[102844]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:54:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:20.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:20.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:20.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:22] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T22:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:22 smithi196 ceph-mon[76349]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:54:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:22 smithi184 ceph-mon[100622]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:54:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:22 smithi184 ceph-mon[102844]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T22:54:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:24 smithi184 ceph-mon[100622]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:54:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:24 smithi184 ceph-mon[102844]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:54:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:24 smithi196 ceph-mon[76349]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T22:54:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:26 smithi196 ceph-mon[76349]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:54:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:26 smithi184 ceph-mon[100622]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:54:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:26 smithi184 ceph-mon[102844]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:54:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:28 smithi196 ceph-mon[76349]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:54:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:28 smithi184 ceph-mon[100622]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:54:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:28 smithi184 ceph-mon[102844]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:54:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:30 smithi196 ceph-mon[76349]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:54:30.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:30 smithi184 ceph-mon[100622]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:54:30.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:30 smithi184 ceph-mon[102844]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:54:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:30.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:30.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:30.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:32.214 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:54:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:54:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 0s ago 3h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 19s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 0s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 0s ago 3h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 0s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 0s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 0s ago 3h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 3h 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 3h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 3h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (2h) 19s ago 3h 80.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:54:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:54:32.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:54:32.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:54:32.743 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:54:32.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:32 smithi196 ceph-mon[76349]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:54:32.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:32.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:32.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[102844]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:54:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[100622]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:54:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:54:33.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:54:33.053 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:54:33.322 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:54:33.323 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:54:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:54:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:33.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:33.637 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:33 smithi196 ceph-mon[76349]: from='client.56385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:33 smithi196 ceph-mon[76349]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:33 smithi196 ceph-mon[76349]: from='client.66359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2525312866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[100622]: from='client.56385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[100622]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[100622]: from='client.66359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2525312866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[102844]: from='client.56385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[102844]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[102844]: from='client.66359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2525312866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: from='client.56409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/338603156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: from='client.56409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/338603156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: from='client.56409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/338603156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:35.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:35.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:36 smithi196 ceph-mon[76349]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:54:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:36 smithi184 ceph-mon[100622]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:54:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:36 smithi184 ceph-mon[102844]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T22:54:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:38 smithi184 ceph-mon[100622]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:54:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:38 smithi184 ceph-mon[102844]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:54:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:38 smithi196 ceph-mon[76349]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:54:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:40 smithi196 ceph-mon[76349]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:54:40.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:40 smithi184 ceph-mon[100622]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:54:40.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:40 smithi184 ceph-mon[102844]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:54:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:40.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:40.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:40.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:54:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:42 smithi196 ceph-mon[76349]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:54:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:42 smithi184 ceph-mon[100622]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:54:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:42 smithi184 ceph-mon[102844]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T22:54:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:44 smithi184 ceph-mon[100622]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:54:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:44 smithi184 ceph-mon[102844]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:54:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:44 smithi196 ceph-mon[76349]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T22:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:45.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:46 smithi196 ceph-mon[76349]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:46 smithi184 ceph-mon[100622]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:46 smithi184 ceph-mon[102844]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:48 smithi196 ceph-mon[76349]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:48 smithi184 ceph-mon[100622]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:48 smithi184 ceph-mon[102844]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:50 smithi196 ceph-mon[76349]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:50.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:50 smithi184 ceph-mon[100622]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:50.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:50 smithi184 ceph-mon[102844]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:50.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:50.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:54:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:54:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:54:50.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:54:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:54:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:54:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:52 smithi196 ceph-mon[76349]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:52.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[100622]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[102844]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[100622]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[102844]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:54 smithi196 ceph-mon[76349]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:54:56.065 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:56.066 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:56 smithi184 ceph-mon[100622]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:56 smithi184 ceph-mon[102844]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:56 smithi196 ceph-mon[76349]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:58 smithi184 ceph-mon[100622]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:58 smithi184 ceph-mon[102844]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:58 smithi196 ceph-mon[76349]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:54:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:54:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:54:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:00 smithi196 ceph-mon[76349]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:00 smithi184 ceph-mon[100622]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:00 smithi184 ceph-mon[102844]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:00.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:00.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:00.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T22:55:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:02 smithi196 ceph-mon[76349]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:02 smithi184 ceph-mon[100622]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:02 smithi184 ceph-mon[102844]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.919 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:55:04.446 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:55:04.446 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 3h - - 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 3h - - 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 12s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 10s ago 3h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (2h) 12s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (2h) 10s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 12s ago 3h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 12s ago 3h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 10s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 12s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 12s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 10s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 12s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 12s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 12s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:55:04.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 12s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 10s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 10s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 10s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 10s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 3h 80.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 10s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 12s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:55:04.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 10s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:55:04.758 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:55:04.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:55:04.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:55:04.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[100622]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[100622]: from='client.66383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/777168287' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[102844]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[102844]: from='client.66383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:05.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/777168287' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:04 smithi196 ceph-mon[76349]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:04 smithi196 ceph-mon[76349]: from='client.66383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/777168287' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:05.341 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:55:05.341 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:05.341 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:05 smithi196 ceph-mon[76349]: from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:05 smithi196 ceph-mon[76349]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/817602115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[100622]: from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[100622]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/817602115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[102844]: from='client.66389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[102844]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/817602115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:06 smithi196 ceph-mon[76349]: from='client.66407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:06 smithi196 ceph-mon[76349]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:06 smithi184 ceph-mon[100622]: from='client.66407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:06 smithi184 ceph-mon[100622]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:06 smithi184 ceph-mon[102844]: from='client.66407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:06 smithi184 ceph-mon[102844]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:08 smithi196 ceph-mon[76349]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:08 smithi184 ceph-mon[100622]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:08 smithi184 ceph-mon[102844]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:10 smithi196 ceph-mon[76349]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:10 smithi184 ceph-mon[100622]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:10 smithi184 ceph-mon[102844]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:10.905Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:10.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:10.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:12.296 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-20T22:55:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:12 smithi196 ceph-mon[76349]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[100622]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[102844]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:14 smithi196 ceph-mon[76349]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[100622]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[102844]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:16.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:16.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:16 smithi196 ceph-mon[76349]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:16 smithi184 ceph-mon[100622]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:16 smithi184 ceph-mon[102844]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:18 smithi196 ceph-mon[76349]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:18 smithi184 ceph-mon[102844]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:18 smithi184 ceph-mon[100622]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:20 smithi196 ceph-mon[76349]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:20 smithi184 ceph-mon[102844]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:20 smithi184 ceph-mon[100622]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:20.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:20.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:20.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:55:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:22 smithi196 ceph-mon[76349]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:22 smithi184 ceph-mon[100622]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:22 smithi184 ceph-mon[102844]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:24 smithi196 ceph-mon[76349]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:24 smithi184 ceph-mon[100622]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:24 smithi184 ceph-mon[102844]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:26 smithi196 ceph-mon[76349]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:26 smithi184 ceph-mon[100622]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:26 smithi184 ceph-mon[102844]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:29.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:28 smithi196 ceph-mon[76349]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:28 smithi184 ceph-mon[100622]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:28 smithi184 ceph-mon[102844]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:30 smithi196 ceph-mon[76349]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:30 smithi184 ceph-mon[100622]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:30 smithi184 ceph-mon[102844]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:30.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:30.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:30.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:55:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[100622]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[102844]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:32 smithi196 ceph-mon[76349]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:33.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:33.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:34 smithi196 ceph-mon[76349]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[100622]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[102844]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:35.639 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:55:36.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:36.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:55:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T22:55:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 3h - - 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (2h) 3s ago 3h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 2s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 3s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 3s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 3s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:55:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 3s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 3s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 83.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:55:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:55:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:55:36.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:55:36.497 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:55:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:55:36.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:55:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[102844]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[102844]: from='client.56457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[102844]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/757983609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[100622]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[100622]: from='client.56457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[100622]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/757983609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:37.079 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:55:37.080 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:37.080 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:37.080 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:37.080 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:36 smithi196 ceph-mon[76349]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:36 smithi196 ceph-mon[76349]: from='client.56457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:36 smithi196 ceph-mon[76349]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/757983609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[100622]: from='client.66431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[100622]: from='client.66443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1744030914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[102844]: from='client.66431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[102844]: from='client.66443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1744030914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:37 smithi196 ceph-mon[76349]: from='client.66431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:37 smithi196 ceph-mon[76349]: from='client.66443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1744030914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:38 smithi184 ceph-mon[100622]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:38 smithi184 ceph-mon[102844]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:38 smithi196 ceph-mon[76349]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:40 smithi184 ceph-mon[100622]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:40 smithi184 ceph-mon[102844]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:40.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:40.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:40.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:40 smithi196 ceph-mon[76349]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:55:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:42 smithi184 ceph-mon[100622]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:42 smithi184 ceph-mon[102844]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:42 smithi196 ceph-mon[76349]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:44 smithi184 ceph-mon[100622]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:44 smithi184 ceph-mon[102844]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:44 smithi196 ceph-mon[76349]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:46 smithi184 ceph-mon[102844]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:46 smithi184 ceph-mon[100622]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:46 smithi196 ceph-mon[76349]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:48 smithi184 ceph-mon[100622]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:48 smithi184 ceph-mon[102844]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:48 smithi196 ceph-mon[76349]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:50 smithi184 ceph-mon[100622]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:50 smithi184 ceph-mon[102844]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:50.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:50.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:55:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:55:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:55:50.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:55:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:50 smithi196 ceph-mon[76349]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:55:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:55:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:52 smithi196 ceph-mon[76349]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[102844]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[100622]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:54 smithi196 ceph-mon[76349]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:55.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[100622]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[102844]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:55:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:57.079 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:56 smithi184 ceph-mon[100622]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:57.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:56 smithi184 ceph-mon[102844]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:56 smithi196 ceph-mon[76349]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:58 smithi184 ceph-mon[100622]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:55:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:58 smithi184 ceph-mon[102844]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:55:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:58 smithi196 ceph-mon[76349]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:55:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:00 smithi184 ceph-mon[100622]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:00 smithi184 ceph-mon[102844]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:00.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:00.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:00.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:00 smithi196 ceph-mon[76349]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:56:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:02 smithi184 ceph-mon[100622]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:02 smithi184 ceph-mon[102844]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:02 smithi196 ceph-mon[76349]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:04 smithi184 ceph-mon[100622]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:04 smithi184 ceph-mon[102844]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:04 smithi196 ceph-mon[76349]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:06 smithi184 ceph-mon[100622]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:06 smithi184 ceph-mon[102844]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:07.381 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:56:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:06 smithi196 ceph-mon[76349]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 3h - - 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 15s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 15s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 15s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 15s ago 3h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (2h) 13s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:56:07.909 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (2h) 15s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 15s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 15s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 15s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 15s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 15s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 13s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 13s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 13s ago 3h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 13s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 3h 83.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 13s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:56:07.910 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 15s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:56:07.911 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 13s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:56:08.217 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:56:08.217 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:56:08.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:56:08.219 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:56:08.485 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:56:08.486 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:56:08.487 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:56:08.795 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:56:08.795 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:08.795 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:08.795 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:08.796 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: from='client.66461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: from='client.66467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/539726433' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3174685476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: from='client.66461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: from='client.66467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/539726433' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3174685476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: from='client.66461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: from='client.66467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/539726433' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3174685476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:09 smithi184 ceph-mon[102844]: from='client.66479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:09 smithi184 ceph-mon[100622]: from='client.66479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:09 smithi196 ceph-mon[76349]: from='client.66479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:10 smithi184 ceph-mon[100622]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:10 smithi184 ceph-mon[102844]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:10.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:10.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:10.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:10 smithi196 ceph-mon[76349]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:12.261 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:56:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[100622]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[102844]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:12 smithi196 ceph-mon[76349]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:13.354 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:14.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[100622]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[102844]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:14 smithi196 ceph-mon[76349]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:15.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:15.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:17 smithi184 ceph-mon[100622]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:17 smithi184 ceph-mon[102844]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:17 smithi196 ceph-mon[76349]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:19 smithi184 ceph-mon[100622]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:19 smithi184 ceph-mon[102844]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:19 smithi196 ceph-mon[76349]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:21 smithi184 ceph-mon[100622]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:21 smithi184 ceph-mon[102844]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:20.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:20.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:20.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:21 smithi196 ceph-mon[76349]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T22:56:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:23 smithi184 ceph-mon[100622]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:23 smithi184 ceph-mon[102844]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:23 smithi196 ceph-mon[76349]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:25 smithi184 ceph-mon[100622]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:25 smithi184 ceph-mon[102844]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:25 smithi196 ceph-mon[76349]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:26.363 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:27 smithi184 ceph-mon[100622]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:27 smithi184 ceph-mon[102844]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:27 smithi196 ceph-mon[76349]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:28 smithi196 ceph-mon[76349]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:28 smithi184 ceph-mon[100622]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:28 smithi184 ceph-mon[102844]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:30 smithi196 ceph-mon[76349]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:30 smithi184 ceph-mon[100622]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:30 smithi184 ceph-mon[102844]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:30.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:30.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:30.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:32 smithi184 ceph-mon[100622]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:32 smithi184 ceph-mon[102844]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:56:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:32 smithi196 ceph-mon[76349]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:34.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[100622]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[102844]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:34 smithi196 ceph-mon[76349]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:36.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:36 smithi184 ceph-mon[100622]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:36 smithi184 ceph-mon[102844]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:36 smithi196 ceph-mon[76349]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:38 smithi184 ceph-mon[100622]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:38 smithi184 ceph-mon[102844]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:39.093 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:56:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:38 smithi196 ceph-mon[76349]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:39.639 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 3h - - 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 3h - - 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 6s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 6s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 6s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 6s ago 3h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 3h 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 6s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 6s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:56:39.640 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 6s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 6s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 6s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 6s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 83.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 6s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:56:39.641 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:56:39.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:56:39.967 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:56:39.967 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:56:39.968 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:56:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:56:40.262 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:56:40.578 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:56:40.578 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:40.578 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:40.578 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:40.578 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:40.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: from='client.56529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/275609114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/969121664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: from='client.56529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/275609114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:40.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/969121664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: from='client.56529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/275609114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:41.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/969121664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:40.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:40.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:40.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:41 smithi196 ceph-mon[76349]: from='client.66515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:41 smithi184 ceph-mon[100622]: from='client.66515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:41 smithi184 ceph-mon[102844]: from='client.66515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:56:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:42 smithi196 ceph-mon[76349]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:42 smithi184 ceph-mon[100622]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:42 smithi184 ceph-mon[102844]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:44 smithi184 ceph-mon[100622]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:44 smithi184 ceph-mon[102844]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:44 smithi196 ceph-mon[76349]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:46.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:46 smithi196 ceph-mon[76349]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:46 smithi184 ceph-mon[100622]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:46 smithi184 ceph-mon[102844]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:48 smithi196 ceph-mon[76349]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:48 smithi184 ceph-mon[100622]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:48 smithi184 ceph-mon[102844]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:50 smithi196 ceph-mon[76349]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:50 smithi184 ceph-mon[100622]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:50 smithi184 ceph-mon[102844]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:50.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:50.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:56:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:56:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:56:50.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:56:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:56:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:56:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:52 smithi196 ceph-mon[76349]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:52 smithi184 ceph-mon[100622]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:52 smithi184 ceph-mon[102844]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:54 smithi196 ceph-mon[76349]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[100622]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[102844]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:56:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:57.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:56 smithi196 ceph-mon[76349]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:56 smithi184 ceph-mon[100622]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:56 smithi184 ceph-mon[102844]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:59.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:58 smithi184 ceph-mon[100622]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:59.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:56:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:59.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:58 smithi184 ceph-mon[102844]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:59.121 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:56:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:58 smithi196 ceph-mon[76349]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:56:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:00.933 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:00.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:00.933 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:00.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:00.933 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:00.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:00 smithi184 ceph-mon[100622]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:00 smithi184 ceph-mon[102844]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:00 smithi196 ceph-mon[76349]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:57:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:02 smithi184 ceph-mon[100622]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:02 smithi184 ceph-mon[102844]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:02 smithi196 ceph-mon[76349]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:04 smithi184 ceph-mon[100622]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:04 smithi184 ceph-mon[102844]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:04 smithi196 ceph-mon[76349]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:06 smithi184 ceph-mon[100622]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:06 smithi184 ceph-mon[102844]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:06 smithi196 ceph-mon[76349]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:08 smithi184 ceph-mon[100622]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:08 smithi184 ceph-mon[102844]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:08 smithi196 ceph-mon[76349]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:10.871 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:57:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:10 smithi184 ceph-mon[100622]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:10 smithi184 ceph-mon[102844]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:10.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:10.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:10.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:10 smithi196 ceph-mon[76349]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:11.404 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:57:11.404 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T22:57:11.404 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 18s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 18s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 18s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 3h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 18s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 18s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 18s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 18s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:57:11.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 18s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 16s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 16s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 16s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 16s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 16s ago 3h 83.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 16s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 18s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:57:11.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 16s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:57:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:57:11.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:57:11.986 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:57:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:57:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:57:11.987 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:57:11.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:11 smithi184 ceph-mon[100622]: from='client.56565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:11.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2813759702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:11.988 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:11 smithi184 ceph-mon[102844]: from='client.56565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:11.988 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2813759702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:12.294 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:57:12.295 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:12.295 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:12.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T22:57:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:11 smithi196 ceph-mon[76349]: from='client.56565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2813759702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[100622]: from='client.66533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[100622]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[100622]: from='client.56577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[100622]: from='client.66551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2126258830' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[102844]: from='client.66533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[102844]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[102844]: from='client.56577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[102844]: from='client.66551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2126258830' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:12 smithi196 ceph-mon[76349]: from='client.66533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:12 smithi196 ceph-mon[76349]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:12 smithi196 ceph-mon[76349]: from='client.56577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:12 smithi196 ceph-mon[76349]: from='client.66551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2126258830' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:14 smithi196 ceph-mon[76349]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:14.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[100622]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[102844]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:15.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:15.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:15.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:15.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:16.759 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:16 smithi196 ceph-mon[76349]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:16 smithi184 ceph-mon[100622]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:16 smithi184 ceph-mon[102844]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:18 smithi184 ceph-mon[100622]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:18 smithi184 ceph-mon[102844]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:18 smithi196 ceph-mon[76349]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:20.901 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:20 smithi184 ceph-mon[100622]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:20.902 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:20 smithi184 ceph-mon[102844]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:20 smithi196 ceph-mon[76349]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:20.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:20.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:20.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:57:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:22 smithi184 ceph-mon[100622]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:22 smithi184 ceph-mon[102844]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:22 smithi196 ceph-mon[76349]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:24 smithi184 ceph-mon[100622]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:24 smithi184 ceph-mon[102844]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:24 smithi196 ceph-mon[76349]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:26 smithi184 ceph-mon[102844]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:26 smithi184 ceph-mon[100622]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:26 smithi196 ceph-mon[76349]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:28 smithi184 ceph-mon[100622]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:28 smithi184 ceph-mon[102844]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:28 smithi196 ceph-mon[76349]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:30.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:30.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:30.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:30 smithi184 ceph-mon[102844]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:31.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:30 smithi184 ceph-mon[100622]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:31.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:30.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:31.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:30.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:31.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:30.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:30 smithi196 ceph-mon[76349]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:57:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:32 smithi184 ceph-mon[100622]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:32 smithi184 ceph-mon[102844]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:32 smithi196 ceph-mon[76349]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[100622]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[102844]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:34 smithi196 ceph-mon[76349]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:36 smithi196 ceph-mon[76349]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:36 smithi184 ceph-mon[100622]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:36 smithi184 ceph-mon[102844]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:38 smithi196 ceph-mon[76349]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:38 smithi184 ceph-mon[100622]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:38 smithi184 ceph-mon[102844]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:40.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:40 smithi184 ceph-mon[100622]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:40.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:40 smithi184 ceph-mon[102844]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:40.998 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:40.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:40.998 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:40.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:40.998 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:40.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:40 smithi196 ceph-mon[76349]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:57:42.580 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:57:43.120 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:42 smithi184 ceph-mon[100622]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:42 smithi184 ceph-mon[102844]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 8s ago 3h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 9s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:57:43.124 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 8s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 9s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 9s ago 3h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 9s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 8s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2h) 9s ago 3h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 9s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 9s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 8s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:57:43.125 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:57:43.126 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 8s ago 3h 83.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:57:43.126 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:57:43.126 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:57:43.126 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:57:43.126 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:57:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:42 smithi196 ceph-mon[76349]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:57:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:57:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:57:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:57:43.449 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:57:43.718 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:57:43.718 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:57:43.718 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:57:43.719 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:57:44.031 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:57:44.032 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:44.032 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:44.032 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:44.032 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[100622]: from='client.56601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[100622]: from='client.56607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3231797972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[102844]: from='client.56601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[102844]: from='client.56607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3231797972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:43 smithi196 ceph-mon[76349]: from='client.56601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:43 smithi196 ceph-mon[76349]: from='client.56607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3231797972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[100622]: from='client.66575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[100622]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[100622]: from='client.66587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/300980869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[102844]: from='client.66575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[102844]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[102844]: from='client.66587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/300980869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:44 smithi196 ceph-mon[76349]: from='client.66575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:44 smithi196 ceph-mon[76349]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:44 smithi196 ceph-mon[76349]: from='client.66587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/300980869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:46 smithi196 ceph-mon[76349]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:46 smithi184 ceph-mon[100622]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:46 smithi184 ceph-mon[102844]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:48 smithi196 ceph-mon[76349]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:48 smithi184 ceph-mon[100622]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:48 smithi184 ceph-mon[102844]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:50 smithi196 ceph-mon[76349]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:50 smithi184 ceph-mon[100622]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:50 smithi184 ceph-mon[102844]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:50.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:50.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:57:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:57:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:57:50.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:57:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:57:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:57:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:52 smithi184 ceph-mon[100622]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:52 smithi184 ceph-mon[102844]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:52 smithi196 ceph-mon[76349]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:54.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:54 smithi196 ceph-mon[76349]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.677 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[102844]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:54.677 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[100622]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:54.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:57:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:57 smithi184 ceph-mon[100622]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:57 smithi184 ceph-mon[102844]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:57 smithi196 ceph-mon[76349]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:59 smithi184 ceph-mon[100622]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:57:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:59 smithi184 ceph-mon[102844]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:57:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:59 smithi196 ceph-mon[76349]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:57:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:01.176 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:01 smithi184 ceph-mon[100622]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:01.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:01 smithi184 ceph-mon[102844]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:01.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:00.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:01.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:00.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:01.177 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:00.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:01.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:01 smithi196 ceph-mon[76349]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:58:03.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:03 smithi184 ceph-mon[100622]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:03.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:03 smithi184 ceph-mon[102844]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:03 smithi196 ceph-mon[76349]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:05 smithi184 ceph-mon[100622]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:05 smithi184 ceph-mon[102844]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:05 smithi196 ceph-mon[76349]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:07 smithi184 ceph-mon[100622]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:07 smithi184 ceph-mon[102844]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:07 smithi196 ceph-mon[76349]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:09 smithi184 ceph-mon[100622]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:09 smithi184 ceph-mon[102844]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:09 smithi196 ceph-mon[76349]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:11 smithi184 ceph-mon[100622]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:11 smithi184 ceph-mon[102844]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:10.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:10.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:10.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:11 smithi196 ceph-mon[76349]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T22:58:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:13 smithi184 ceph-mon[100622]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:13 smithi184 ceph-mon[102844]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:13 smithi196 ceph-mon[76349]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 22:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T22:58:13.810983775Z level=info msg="Completed cleanup jobs" duration=11.540664ms 2024-09-20T22:58:14.321 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[102844]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[100622]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:14.851 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:58:14.851 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 3h - - 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 1s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 1s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 1s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 1s ago 3h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:58:14.852 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 19s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 1s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 1s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (2h) 1s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 1s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 1s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 19s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 19s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:58:14.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 19s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 19s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 19s ago 3h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 19s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 1s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:58:14.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 19s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:14 smithi196 ceph-mon[76349]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:15.168 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:58:15.168 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:58:15.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:58:15.169 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:58:15.441 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:58:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:58:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:58:15.442 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:58:15.753 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:58:15.753 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:15.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:15.753 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:15.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:15.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='client.56637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='client.56649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3328928960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='client.56637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='client.56649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3328928960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='client.56637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='client.56649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3328928960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[100622]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[100622]: from='client.56661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/696975358' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[102844]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[102844]: from='client.56661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/696975358' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:16 smithi196 ceph-mon[76349]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:16 smithi196 ceph-mon[76349]: from='client.56661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/696975358' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:18 smithi196 ceph-mon[76349]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:18 smithi184 ceph-mon[100622]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:18 smithi184 ceph-mon[102844]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:20 smithi184 ceph-mon[100622]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:20 smithi184 ceph-mon[102844]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:20 smithi196 ceph-mon[76349]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:20.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:20.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:20.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:22.225 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:22 smithi184 ceph-mon[100622]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:22 smithi184 ceph-mon[102844]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:22 smithi196 ceph-mon[76349]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:24.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:24 smithi184 ceph-mon[100622]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:24.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:24 smithi184 ceph-mon[102844]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:24 smithi196 ceph-mon[76349]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:25.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:26 smithi184 ceph-mon[100622]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:26 smithi184 ceph-mon[102844]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:26 smithi196 ceph-mon[76349]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:28 smithi196 ceph-mon[76349]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:28 smithi184 ceph-mon[100622]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:28 smithi184 ceph-mon[102844]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:30 smithi196 ceph-mon[76349]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:30.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:30 smithi184 ceph-mon[100622]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:30.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:30 smithi184 ceph-mon[102844]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:30.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:30.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:30.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:58:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:32 smithi196 ceph-mon[76349]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:32 smithi184 ceph-mon[102844]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:32 smithi184 ceph-mon[100622]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:34 smithi196 ceph-mon[76349]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[100622]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[102844]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:36 smithi196 ceph-mon[76349]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:36 smithi184 ceph-mon[102844]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:36 smithi184 ceph-mon[100622]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:38.824 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:38 smithi196 ceph-mon[76349]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:38 smithi184 ceph-mon[102844]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:38 smithi184 ceph-mon[100622]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:58:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:40 smithi196 ceph-mon[76349]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:40.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:40 smithi184 ceph-mon[100622]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:40.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:40 smithi184 ceph-mon[102844]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:40.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:40.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:40.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T22:58:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:42 smithi196 ceph-mon[76349]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:42.901 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:42 smithi184 ceph-mon[100622]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:42.902 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:42 smithi184 ceph-mon[102844]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:44 smithi196 ceph-mon[76349]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:44.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:44 smithi184 ceph-mon[100622]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:44.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:44.907 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:44 smithi184 ceph-mon[102844]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:44.907 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:45.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:46.036 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:58:46.559 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 3h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (2h) 13s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 13s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 11s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:58:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 11s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:58:46.561 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:58:46.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:46 smithi184 ceph-mon[100622]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:46.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:46 smithi184 ceph-mon[100622]: from='client.56673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:46.872 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:46 smithi184 ceph-mon[102844]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:46.872 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:46 smithi184 ceph-mon[102844]: from='client.56673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:58:46.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:58:46.874 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:58:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:46 smithi196 ceph-mon[76349]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:46 smithi196 ceph-mon[76349]: from='client.56673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.141 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:58:47.141 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:58:47.141 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:58:47.141 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:58:47.142 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:58:47.452 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:58:47.452 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:47.453 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:47.453 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:47.453 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[100622]: from='client.56679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[100622]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2977347583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2367613577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[102844]: from='client.56679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[102844]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2977347583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2367613577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:47 smithi196 ceph-mon[76349]: from='client.56679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:47 smithi196 ceph-mon[76349]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2977347583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2367613577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:48 smithi196 ceph-mon[76349]: from='client.66659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:48 smithi196 ceph-mon[76349]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:48 smithi184 ceph-mon[100622]: from='client.66659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:48 smithi184 ceph-mon[100622]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:48 smithi184 ceph-mon[102844]: from='client.66659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:48 smithi184 ceph-mon[102844]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:50 smithi196 ceph-mon[76349]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:50.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:50 smithi184 ceph-mon[100622]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:50.925 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:50 smithi184 ceph-mon[102844]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:50.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:50.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:58:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:58:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:58:50.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:58:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:58:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:58:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:52 smithi196 ceph-mon[76349]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:52 smithi184 ceph-mon[100622]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:52 smithi184 ceph-mon[102844]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[100622]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[102844]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:54 smithi196 ceph-mon[76349]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:58:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:56 smithi184 ceph-mon[100622]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:56 smithi184 ceph-mon[102844]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:56 smithi196 ceph-mon[76349]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:58.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:58 smithi196 ceph-mon[76349]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:58 smithi184 ceph-mon[100622]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:58 smithi184 ceph-mon[102844]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:59.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:00 smithi196 ceph-mon[76349]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:00 smithi184 ceph-mon[100622]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:00 smithi184 ceph-mon[102844]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:00.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:00.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:01.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:00.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:59:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:02 smithi184 ceph-mon[100622]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:02 smithi184 ceph-mon[102844]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:02 smithi196 ceph-mon[76349]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:04 smithi184 ceph-mon[100622]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:04 smithi184 ceph-mon[102844]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:04 smithi196 ceph-mon[76349]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:05.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:05.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:06 smithi184 ceph-mon[100622]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:06 smithi184 ceph-mon[102844]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:06 smithi196 ceph-mon[76349]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:08 smithi184 ceph-mon[100622]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:08 smithi184 ceph-mon[102844]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:08 smithi196 ceph-mon[76349]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:10 smithi184 ceph-mon[100622]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:10 smithi184 ceph-mon[102844]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:10.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:10.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:10.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:10 smithi196 ceph-mon[76349]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T22:59:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:12 smithi184 ceph-mon[100622]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:12 smithi184 ceph-mon[102844]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:12 smithi196 ceph-mon[76349]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[100622]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[102844]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:14 smithi196 ceph-mon[76349]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:16 smithi196 ceph-mon[76349]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:16 smithi184 ceph-mon[102844]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:16 smithi184 ceph-mon[100622]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.748 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 3h - - 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 3h - - 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 4s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:59:18.273 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 4s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 4s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 4s ago 3h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 2s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 4s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 4s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 4s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 3h 70.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (2h) 4s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 2s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 2s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 2s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:59:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 2s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:59:18.275 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 80.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:59:18.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:59:18.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 2s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:59:18.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 4s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:59:18.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 2s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:59:18.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:59:18.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:59:18.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:59:18.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:59:18.853 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:59:18.853 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:59:18.854 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[100622]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[100622]: from='client.56709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[100622]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2171667740' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[102844]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[102844]: from='client.56709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[102844]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:18.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2171667740' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:19.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:18 smithi196 ceph-mon[76349]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:19.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:18 smithi196 ceph-mon[76349]: from='client.56709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:19.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:18 smithi196 ceph-mon[76349]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:19.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2171667740' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:19.163 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:59:19.163 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:19.164 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:19.164 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:19.164 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:20.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:19 smithi196 ceph-mon[76349]: from='client.66683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:19 smithi196 ceph-mon[76349]: from='client.66695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/632510241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[100622]: from='client.66683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[100622]: from='client.66695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/632510241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[102844]: from='client.66683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[102844]: from='client.66695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/632510241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:20 smithi196 ceph-mon[76349]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:20 smithi184 ceph-mon[100622]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:20 smithi184 ceph-mon[102844]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:20.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:20.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:20.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:59:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:22 smithi196 ceph-mon[76349]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:22 smithi184 ceph-mon[100622]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:22 smithi184 ceph-mon[102844]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:25.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:24 smithi184 ceph-mon[100622]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:25.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:24 smithi184 ceph-mon[102844]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:24 smithi196 ceph-mon[76349]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:25.952 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:25.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:27.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:26 smithi184 ceph-mon[100622]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:27.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:26 smithi184 ceph-mon[102844]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:26 smithi196 ceph-mon[76349]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:28 smithi196 ceph-mon[76349]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:28 smithi184 ceph-mon[100622]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:28 smithi184 ceph-mon[102844]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:30 smithi196 ceph-mon[76349]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:30 smithi184 ceph-mon[100622]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:30 smithi184 ceph-mon[102844]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:30.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:30.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:30.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:59:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:32 smithi196 ceph-mon[76349]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:32 smithi184 ceph-mon[100622]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:32 smithi184 ceph-mon[102844]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[100622]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[102844]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:34 smithi196 ceph-mon[76349]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:36 smithi184 ceph-mon[100622]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:36 smithi184 ceph-mon[102844]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:36 smithi196 ceph-mon[76349]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:38 smithi184 ceph-mon[100622]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:38 smithi184 ceph-mon[102844]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:39.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:38 smithi196 ceph-mon[76349]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:40 smithi196 ceph-mon[76349]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:40 smithi184 ceph-mon[100622]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:40 smithi184 ceph-mon[102844]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:40.928Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:40.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:40.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T22:59:43.249 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:42 smithi184 ceph-mon[100622]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:43.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:42 smithi184 ceph-mon[102844]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:42 smithi196 ceph-mon[76349]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:45.034 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:44 smithi184 ceph-mon[100622]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:45.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:44 smithi184 ceph-mon[102844]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:44 smithi196 ceph-mon[76349]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:46.113 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:46.113 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:47.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:46 smithi184 ceph-mon[100622]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:47.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:46 smithi184 ceph-mon[102844]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:46 smithi196 ceph-mon[76349]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:48 smithi184 ceph-mon[100622]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:48 smithi184 ceph-mon[102844]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:48 smithi196 ceph-mon[76349]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:49.459 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T22:59:49.986 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:59:49.986 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T22:59:49.986 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T22:59:49.986 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T22:59:49.986 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 3h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 14s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T22:59:49.987 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 14s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 14s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 80.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T22:59:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T22:59:50.296 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:59:50.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T22:59:50.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:59:50.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:50.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T22:59:50.297 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:59:50.565 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:59:50.566 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T22:59:50.567 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T22:59:50.872 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:59:50.872 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:50.872 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:50.872 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:50.872 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: from='client.56745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: from='client.66713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: from='client.66719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3735977292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3708169084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: from='client.56745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: from='client.66713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: from='client.66719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3735977292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:51.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3708169084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:50.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:50.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T22:59:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 22:59:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T22:59:50.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: from='client.56745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: from='client.66713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: from='client.66719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3735977292' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3708169084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:51 smithi184 ceph-mon[100622]: from='client.66731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:51 smithi184 ceph-mon[102844]: from='client.66731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 22:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:22:59:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T22:59:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:51 smithi196 ceph-mon[76349]: from='client.66731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:52 smithi184 ceph-mon[100622]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:52 smithi184 ceph-mon[102844]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:52 smithi196 ceph-mon[76349]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[100622]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[102844]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:54 smithi196 ceph-mon[76349]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T22:59:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:57 smithi184 ceph-mon[100622]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:57 smithi184 ceph-mon[102844]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:57 smithi196 ceph-mon[76349]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:59 smithi184 ceph-mon[100622]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 22:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:59 smithi184 ceph-mon[102844]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 22:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:59 smithi196 ceph-mon[76349]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 22:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T22:59:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:01 smithi196 ceph-mon[76349]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:01 smithi184 ceph-mon[100622]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:01 smithi184 ceph-mon[102844]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:00.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:00.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:00.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:02.297 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 23:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T23:00:02.045Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726862400138 maxt=1726869600000 ulid=01J88TKVAM5VT5Y1BAKAMS4YSM duration=41.217699ms 2024-09-20T23:00:02.298 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 20 23:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-20T23:00:02.048Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.195877ms 2024-09-20T23:00:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:00:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:03 smithi184 ceph-mon[100622]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:00:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:03 smithi184 ceph-mon[102844]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:00:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:03 smithi196 ceph-mon[76349]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:00:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:05 smithi184 ceph-mon[100622]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:05 smithi184 ceph-mon[102844]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:05 smithi196 ceph-mon[76349]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:07.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:07 smithi184 ceph-mon[100622]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:07.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:07 smithi184 ceph-mon[102844]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:07 smithi196 ceph-mon[76349]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:09 smithi184 ceph-mon[100622]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:09 smithi184 ceph-mon[102844]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:09 smithi196 ceph-mon[76349]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:11 smithi184 ceph-mon[100622]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:11 smithi184 ceph-mon[102844]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:10.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:10.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:10.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:11 smithi196 ceph-mon[76349]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:00:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:13 smithi184 ceph-mon[102844]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:13 smithi184 ceph-mon[100622]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:13 smithi196 ceph-mon[76349]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:14.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[100622]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[102844]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:15 smithi196 ceph-mon[76349]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:17 smithi196 ceph-mon[76349]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:17 smithi184 ceph-mon[100622]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:17 smithi184 ceph-mon[102844]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:19 smithi196 ceph-mon[76349]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:19 smithi184 ceph-mon[100622]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:19 smithi184 ceph-mon[102844]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:20 smithi196 ceph-mon[76349]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:20 smithi184 ceph-mon[100622]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:20 smithi184 ceph-mon[102844]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:21.168 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:00:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:20.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:20.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:20.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 3h - - 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:00:21.695 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 3h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 5s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 5s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 5s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 5s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 81.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:00:21.696 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:00:21.697 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 5s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:00:21.697 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:00:21.697 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 5s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:00:22.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:00:22.007 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:00:22.279 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:00:22.280 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:22] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[100622]: from='client.56781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[100622]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[100622]: from='client.66749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[100622]: from='client.66755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/125756788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:22.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[102844]: from='client.56781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[102844]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:22.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[102844]: from='client.66749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[102844]: from='client.66755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/125756788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:22 smithi196 ceph-mon[76349]: from='client.56781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:22 smithi196 ceph-mon[76349]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:22 smithi196 ceph-mon[76349]: from='client.66749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:22 smithi196 ceph-mon[76349]: from='client.66755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:22.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/125756788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:22.591 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:22.591 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:22.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:22.591 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:22.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:23 smithi196 ceph-mon[76349]: from='client.66767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/826812000' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:23 smithi184 ceph-mon[100622]: from='client.66767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/826812000' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:23 smithi184 ceph-mon[102844]: from='client.66767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:24.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/826812000' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:24 smithi196 ceph-mon[76349]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:24 smithi184 ceph-mon[100622]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:24 smithi184 ceph-mon[102844]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:26.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:26 smithi184 ceph-mon[102844]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:26 smithi184 ceph-mon[100622]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:26 smithi196 ceph-mon[76349]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:28 smithi184 ceph-mon[100622]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:28 smithi184 ceph-mon[102844]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:28 smithi196 ceph-mon[76349]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:30 smithi196 ceph-mon[76349]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:30.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:30 smithi184 ceph-mon[100622]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:30.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:30 smithi184 ceph-mon[102844]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:30.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:30.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:30.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:00:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:32 smithi196 ceph-mon[76349]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:32 smithi184 ceph-mon[100622]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:32 smithi184 ceph-mon[102844]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:34 smithi196 ceph-mon[76349]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[102844]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[100622]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[100622]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[102844]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:37 smithi196 ceph-mon[76349]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:39 smithi184 ceph-mon[102844]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:39 smithi184 ceph-mon[100622]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:39 smithi196 ceph-mon[76349]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:41 smithi184 ceph-mon[100622]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:41 smithi184 ceph-mon[102844]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:40.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:40.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:40.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:41 smithi196 ceph-mon[76349]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:00:43.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:43 smithi196 ceph-mon[76349]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:43 smithi184 ceph-mon[100622]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:43 smithi184 ceph-mon[102844]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:45 smithi184 ceph-mon[100622]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:45 smithi184 ceph-mon[102844]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:45 smithi196 ceph-mon[76349]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:47 smithi184 ceph-mon[100622]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:47 smithi184 ceph-mon[102844]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:47 smithi196 ceph-mon[76349]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:49 smithi184 ceph-mon[100622]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:49 smithi184 ceph-mon[102844]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:49 smithi196 ceph-mon[76349]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:51 smithi184 ceph-mon[100622]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:51 smithi184 ceph-mon[102844]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:50.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:50.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:00:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:00:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:00:50.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:00:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:51 smithi196 ceph-mon[76349]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:00:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-20T23:00:52.886 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:00:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:53 smithi184 ceph-mon[100622]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:53 smithi184 ceph-mon[102844]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:53 smithi196 ceph-mon[76349]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.417 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:00:53.417 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 3h - - 2024-09-20T23:00:53.417 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 19s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 3h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 19s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:00:53.418 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (2h) 17s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 17s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 17s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 17s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 107M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 17s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 19s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:00:53.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 17s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:53.730 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:00:53.731 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:00:54.000 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:00:54.001 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:00:54.001 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:00:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:54 smithi184 ceph-mon[100622]: from='client.56817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1206510250' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:54 smithi184 ceph-mon[102844]: from='client.56817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1206510250' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:54.309 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:54.309 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:54.309 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:54.309 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:54.309 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:54 smithi196 ceph-mon[76349]: from='client.56817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1206510250' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='client.66785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='client.66803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2213779032' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='client.66785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='client.66803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2213779032' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='client.66785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='client.66803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2213779032' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:55.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:57 smithi196 ceph-mon[76349]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[100622]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[102844]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:00:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:59 smithi184 ceph-mon[100622]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:59 smithi184 ceph-mon[102844]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:59 smithi196 ceph-mon[76349]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:01 smithi184 ceph-mon[100622]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:01 smithi184 ceph-mon[102844]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:00.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:00.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:00.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:01 smithi196 ceph-mon[76349]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T23:01:03.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:03 smithi196 ceph-mon[76349]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:03 smithi184 ceph-mon[100622]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:03 smithi184 ceph-mon[102844]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:05 smithi184 ceph-mon[100622]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:05 smithi184 ceph-mon[102844]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:05.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:05 smithi196 ceph-mon[76349]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:07 smithi196 ceph-mon[76349]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:07.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:07 smithi184 ceph-mon[100622]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:07.431 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:07 smithi184 ceph-mon[102844]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:08 smithi184 ceph-mon[100622]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:08 smithi184 ceph-mon[102844]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:08 smithi196 ceph-mon[76349]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:10 smithi184 ceph-mon[100622]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:10 smithi184 ceph-mon[102844]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:10 smithi196 ceph-mon[76349]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:10.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:10.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:10.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:12.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T23:01:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:12 smithi184 ceph-mon[100622]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:12 smithi184 ceph-mon[102844]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:12 smithi196 ceph-mon[76349]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:14 smithi184 ceph-mon[100622]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:14 smithi184 ceph-mon[102844]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:14 smithi196 ceph-mon[76349]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:16 smithi196 ceph-mon[76349]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:16.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[100622]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[102844]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:18 smithi196 ceph-mon[76349]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:18 smithi184 ceph-mon[100622]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:18 smithi184 ceph-mon[102844]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:20 smithi196 ceph-mon[76349]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:20.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:20 smithi184 ceph-mon[100622]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:20.937 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:20 smithi184 ceph-mon[102844]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:20.936Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:20.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:20.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:01:22.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:22 smithi196 ceph-mon[76349]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:22 smithi184 ceph-mon[100622]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:22 smithi184 ceph-mon[102844]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:24.607 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:01:24.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:24 smithi196 ceph-mon[76349]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:24.846 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:24 smithi184 ceph-mon[100622]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:24.846 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:24 smithi184 ceph-mon[102844]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:25.135 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 8s ago 3h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 8s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 3h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:01:25.136 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 3h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (2h) 8s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 8s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 8s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 8s ago 3h 107M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 8s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:01:25.137 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 8s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:01:25.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:01:25.718 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:01:25.719 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:01:25.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[100622]: from='client.56853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[100622]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2782395478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[102844]: from='client.56853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[102844]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2782395478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:25 smithi196 ceph-mon[76349]: from='client.56853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:25 smithi196 ceph-mon[76349]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2782395478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:26.029 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:01:26.029 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:26.029 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:26 smithi196 ceph-mon[76349]: from='client.66827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:26 smithi196 ceph-mon[76349]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:26 smithi196 ceph-mon[76349]: from='client.56877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/383831248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[102844]: from='client.66827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[102844]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[102844]: from='client.56877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/383831248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[100622]: from='client.66827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[100622]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[100622]: from='client.56877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/383831248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:28 smithi196 ceph-mon[76349]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:29.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:28 smithi184 ceph-mon[100622]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:29.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:28 smithi184 ceph-mon[102844]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:29.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:29.893 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:30 smithi196 ceph-mon[76349]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:30.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:30 smithi184 ceph-mon[100622]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:30.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:30 smithi184 ceph-mon[102844]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:30.936Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:30.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:30.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:01:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:32 smithi196 ceph-mon[76349]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:32 smithi184 ceph-mon[100622]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:32 smithi184 ceph-mon[102844]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:34 smithi196 ceph-mon[76349]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:34 smithi184 ceph-mon[100622]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:34 smithi184 ceph-mon[102844]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[100622]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[102844]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:36 smithi196 ceph-mon[76349]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:38 smithi184 ceph-mon[100622]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:38 smithi184 ceph-mon[102844]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:38 smithi196 ceph-mon[76349]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:40 smithi184 ceph-mon[100622]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:40 smithi184 ceph-mon[102844]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:40.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:40.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:40.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:40 smithi196 ceph-mon[76349]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:01:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:42 smithi184 ceph-mon[100622]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:42 smithi184 ceph-mon[102844]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:42 smithi196 ceph-mon[76349]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:44.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:44 smithi196 ceph-mon[76349]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:44.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:44 smithi184 ceph-mon[102844]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:44 smithi184 ceph-mon[100622]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:45.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:46 smithi196 ceph-mon[76349]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:46 smithi184 ceph-mon[100622]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:46 smithi184 ceph-mon[102844]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:48.949 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:48 smithi184 ceph-mon[100622]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:48.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:48 smithi184 ceph-mon[102844]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:48 smithi196 ceph-mon[76349]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:50.877 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:50 smithi184 ceph-mon[100622]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:50.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:50 smithi184 ceph-mon[102844]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:50 smithi196 ceph-mon[76349]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:51.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:50.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:51.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:50.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:01:51.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:01:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:01:50.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:01:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:01:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:01:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:52 smithi184 ceph-mon[100622]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:52 smithi184 ceph-mon[102844]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:52 smithi196 ceph-mon[76349]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:54 smithi184 ceph-mon[100622]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:54 smithi184 ceph-mon[102844]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:54 smithi196 ceph-mon[76349]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:56.326 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 2s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 3h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:01:56.861 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 3h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 0s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 3h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 2s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 0s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 3h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 3h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:01:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 0s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 0s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 0s ago 3h 67.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 0s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 2s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:01:56.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 0s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:01:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:56 smithi196 ceph-mon[76349]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:01:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:01:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:01:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:01:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:01:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:01:57.172 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:01:57.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[100622]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:57.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[102844]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:57.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:01:57.439 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:01:57.440 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:01:57.440 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:01:57.440 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:01:57.748 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:01:57.748 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:57.748 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[100622]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[100622]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[100622]: from='client.66863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/321081852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/536598082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[102844]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[102844]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[102844]: from='client.66863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/321081852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:58.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/536598082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:57 smithi196 ceph-mon[76349]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:57 smithi196 ceph-mon[76349]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:57 smithi196 ceph-mon[76349]: from='client.66863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/321081852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/536598082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:58 smithi196 ceph-mon[76349]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:58 smithi196 ceph-mon[76349]: from='client.56913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:58 smithi184 ceph-mon[100622]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:58 smithi184 ceph-mon[100622]: from='client.56913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:58 smithi184 ceph-mon[102844]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:58 smithi184 ceph-mon[102844]: from='client.56913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:00 smithi196 ceph-mon[76349]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:00 smithi184 ceph-mon[100622]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:00 smithi184 ceph-mon[102844]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:00.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:00.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:00.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:02:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:02 smithi196 ceph-mon[76349]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:02 smithi184 ceph-mon[100622]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:02 smithi184 ceph-mon[102844]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:04 smithi196 ceph-mon[76349]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:04 smithi184 ceph-mon[100622]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:04 smithi184 ceph-mon[102844]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:06 smithi196 ceph-mon[76349]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:06 smithi184 ceph-mon[100622]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:06 smithi184 ceph-mon[102844]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:08 smithi196 ceph-mon[76349]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:09.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:08 smithi184 ceph-mon[100622]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:09.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:08 smithi184 ceph-mon[102844]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:11.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:10 smithi184 ceph-mon[100622]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:11.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:10 smithi184 ceph-mon[102844]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:11.129 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:10.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:11.129 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:10.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:11.129 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:10.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:10 smithi196 ceph-mon[76349]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:02:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:12 smithi196 ceph-mon[76349]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:12 smithi184 ceph-mon[100622]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:12 smithi184 ceph-mon[102844]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:14 smithi196 ceph-mon[76349]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:14 smithi184 ceph-mon[100622]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:14 smithi184 ceph-mon[102844]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[100622]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[102844]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:16 smithi196 ceph-mon[76349]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:18 smithi184 ceph-mon[100622]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:18 smithi184 ceph-mon[102844]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:18 smithi196 ceph-mon[76349]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:21.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:20 smithi184 ceph-mon[100622]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:21.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:20 smithi184 ceph-mon[102844]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:21.203 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:20.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:21.203 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:20.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:21.203 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:20.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:20 smithi196 ceph-mon[76349]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:02:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:22 smithi184 ceph-mon[100622]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:22 smithi184 ceph-mon[102844]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:22 smithi196 ceph-mon[76349]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:25.256 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:24 smithi196 ceph-mon[76349]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:24 smithi184 ceph-mon[100622]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:24 smithi184 ceph-mon[102844]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:26.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:26 smithi196 ceph-mon[76349]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:26 smithi184 ceph-mon[100622]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:26 smithi184 ceph-mon[102844]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:28.043 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:02:28.574 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 3h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 13s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 13s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 3h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 3h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 11s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 11s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 68.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:02:28.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:02:28.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 11s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:02:28.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:02:28.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 11s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:02:28.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:02:28.883 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:02:29.166 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:02:29.166 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:02:29.166 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:02:29.166 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:02:29.167 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:02:29.167 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[102844]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[102844]: from='client.56925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/656772187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[100622]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[100622]: from='client.56925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/656772187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:29.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:28 smithi196 ceph-mon[76349]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:28 smithi196 ceph-mon[76349]: from='client.56925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/656772187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.480 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:02:29.481 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:29.481 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:29.481 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:29.481 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:30.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[100622]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[100622]: from='client.66899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3011545844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:30.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[102844]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[102844]: from='client.66899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3011545844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:30 smithi196 ceph-mon[76349]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:30 smithi196 ceph-mon[76349]: from='client.66899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3011545844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:31.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:31 smithi184 ceph-mon[100622]: from='client.66911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:31.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:31 smithi184 ceph-mon[100622]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:31.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:31 smithi184 ceph-mon[102844]: from='client.66911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:31.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:31 smithi184 ceph-mon[102844]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:31.211 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:30.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:31.211 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:30.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:31.211 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:30.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:31 smithi196 ceph-mon[76349]: from='client.66911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:31 smithi196 ceph-mon[76349]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:32.239 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:02:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:33 smithi184 ceph-mon[100622]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.319 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:33 smithi184 ceph-mon[102844]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:33 smithi196 ceph-mon[76349]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:35 smithi184 ceph-mon[100622]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:35 smithi184 ceph-mon[102844]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:35 smithi196 ceph-mon[76349]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:37 smithi196 ceph-mon[76349]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[100622]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[102844]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:39 smithi184 ceph-mon[100622]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:39 smithi184 ceph-mon[102844]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:39 smithi196 ceph-mon[76349]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:41 smithi184 ceph-mon[100622]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:41 smithi184 ceph-mon[102844]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:40.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:40.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:40.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:41 smithi196 ceph-mon[76349]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:02:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:43 smithi184 ceph-mon[100622]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:43 smithi184 ceph-mon[102844]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:43 smithi196 ceph-mon[76349]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:45 smithi184 ceph-mon[100622]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:45 smithi184 ceph-mon[102844]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:45 smithi196 ceph-mon[76349]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:46.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:47 smithi184 ceph-mon[100622]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:47 smithi184 ceph-mon[102844]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:47 smithi196 ceph-mon[76349]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:49 smithi184 ceph-mon[100622]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:49 smithi184 ceph-mon[102844]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:49 smithi196 ceph-mon[76349]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:51.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:51 smithi184 ceph-mon[100622]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:51.279 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:51 smithi184 ceph-mon[102844]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:51.279 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:50.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:51.279 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:50.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:02:51.279 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:02:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:02:50.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:02:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:51 smithi196 ceph-mon[76349]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:02:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:02:53.236 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:53 smithi184 ceph-mon[100622]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.236 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:53 smithi184 ceph-mon[102844]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:53 smithi196 ceph-mon[76349]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:55.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:55 smithi184 ceph-mon[100622]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:55.236 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:55 smithi184 ceph-mon[102844]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:55 smithi196 ceph-mon[76349]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:57 smithi196 ceph-mon[76349]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[102844]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[100622]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:59 smithi196 ceph-mon[76349]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:59 smithi184 ceph-mon[100622]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:59 smithi184 ceph-mon[102844]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 3h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 3s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:03:00.306 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 3s ago 3h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 3h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (2h) 3s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 3s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 3s ago 3h 68.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 3s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 5s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:03:00.307 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 3s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:00.620 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:03:00.621 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:03:00.886 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:03:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:03:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:03:00.887 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:03:01.199 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:03:01.199 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:01.199 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:01.199 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:01.199 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:01.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:00.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:01.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:00.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:01.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:00.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[100622]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[100622]: from='client.66923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[100622]: from='client.66929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2230291056' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[102844]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[102844]: from='client.66923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[102844]: from='client.66929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2230291056' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:01 smithi196 ceph-mon[76349]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:01 smithi196 ceph-mon[76349]: from='client.66923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:01 smithi196 ceph-mon[76349]: from='client.66929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2230291056' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[100622]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[100622]: from='client.56985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/963136981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:02.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:03:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[102844]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[102844]: from='client.56985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/963136981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:02 smithi196 ceph-mon[76349]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:02 smithi196 ceph-mon[76349]: from='client.56985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/963136981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:03 smithi184 ceph-mon[100622]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:03 smithi184 ceph-mon[102844]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:03 smithi196 ceph-mon[76349]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:05 smithi184 ceph-mon[100622]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:05 smithi184 ceph-mon[102844]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:05 smithi196 ceph-mon[76349]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:07 smithi196 ceph-mon[76349]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:07 smithi184 ceph-mon[100622]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:07 smithi184 ceph-mon[102844]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:09.332 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:09 smithi196 ceph-mon[76349]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:09 smithi184 ceph-mon[100622]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:09 smithi184 ceph-mon[102844]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:11 smithi184 ceph-mon[100622]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:11 smithi184 ceph-mon[102844]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:10.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:10.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:11.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:10.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:11 smithi196 ceph-mon[76349]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:03:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:13 smithi196 ceph-mon[76349]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:13.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:13 smithi184 ceph-mon[100622]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:13.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:13 smithi184 ceph-mon[102844]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:15 smithi184 ceph-mon[100622]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:15 smithi184 ceph-mon[102844]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:15 smithi196 ceph-mon[76349]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:16 smithi196 ceph-mon[76349]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[100622]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[102844]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:19 smithi196 ceph-mon[76349]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:19 smithi184 ceph-mon[100622]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:19 smithi184 ceph-mon[102844]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:21 smithi184 ceph-mon[100622]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:21 smithi184 ceph-mon[102844]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:20.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:20.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:20.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:21 smithi196 ceph-mon[76349]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:03:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:23 smithi196 ceph-mon[76349]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:23 smithi184 ceph-mon[100622]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:23 smithi184 ceph-mon[102844]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:25 smithi184 ceph-mon[100622]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:25 smithi184 ceph-mon[102844]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:25 smithi196 ceph-mon[76349]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:27.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:27 smithi196 ceph-mon[76349]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:27 smithi184 ceph-mon[100622]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:27 smithi184 ceph-mon[102844]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:29.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:29 smithi196 ceph-mon[76349]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:29.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:29 smithi184 ceph-mon[100622]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:29.594 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:29.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:29 smithi184 ceph-mon[102844]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:29.594 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:31 smithi184 ceph-mon[100622]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:31 smithi184 ceph-mon[102844]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:30.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:30.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:30.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:31 smithi196 ceph-mon[76349]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:31.496 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:03:32.026 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:03:32.026 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T23:03:32.026 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 3h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 16.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:03:32.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 3h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 3h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (2h) 14s ago 3h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 65.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (2h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 14s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:03:32.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 14s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:03:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:03:32.352 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:03:32.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:03:32.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:03:32.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:03:32.353 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:03:32.626 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:03:32.627 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:03:32.627 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:03:32.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:03:32.939 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:03:32.940 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:32.940 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:32.940 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:32.940 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:33.199 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: from='client.66959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: from='client.57003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: from='client.66971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2230216015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1023346498' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: from='client.66959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: from='client.57003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: from='client.66971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.200 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2230216015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:33.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1023346498' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: from='client.66959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: from='client.57003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: from='client.66971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2230216015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:33.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1023346498' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:34 smithi184 ceph-mon[100622]: from='client.66983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:34 smithi184 ceph-mon[102844]: from='client.66983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:34 smithi196 ceph-mon[76349]: from='client.66983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:35 smithi184 ceph-mon[100622]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:35 smithi184 ceph-mon[102844]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:35 smithi196 ceph-mon[76349]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:37 smithi196 ceph-mon[76349]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:37 smithi184 ceph-mon[100622]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:37 smithi184 ceph-mon[102844]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[100622]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:38 smithi184 ceph-mon[102844]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:38 smithi196 ceph-mon[76349]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:40 smithi196 ceph-mon[76349]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:40 smithi184 ceph-mon[102844]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:40 smithi184 ceph-mon[100622]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:40.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:40.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:40.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:03:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:42 smithi196 ceph-mon[76349]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:42 smithi184 ceph-mon[100622]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:42 smithi184 ceph-mon[102844]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:44 smithi196 ceph-mon[76349]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:44 smithi184 ceph-mon[100622]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:44 smithi184 ceph-mon[102844]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:45.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:46 smithi184 ceph-mon[100622]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:46 smithi184 ceph-mon[102844]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:46 smithi196 ceph-mon[76349]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:48.480 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:48 smithi196 ceph-mon[76349]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:48 smithi184 ceph-mon[100622]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:48 smithi184 ceph-mon[102844]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:50 smithi196 ceph-mon[76349]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:50 smithi184 ceph-mon[100622]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:50 smithi184 ceph-mon[102844]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:50.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:50.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:03:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:03:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:03:50.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:03:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:03:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:03:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:52 smithi196 ceph-mon[76349]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:52.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:52 smithi184 ceph-mon[100622]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:52.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:52 smithi184 ceph-mon[102844]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:54 smithi196 ceph-mon[76349]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:54.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:54 smithi184 ceph-mon[100622]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:54.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:54 smithi184 ceph-mon[102844]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:56 smithi196 ceph-mon[76349]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:56 smithi184 ceph-mon[100622]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:56 smithi184 ceph-mon[102844]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[102844]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[100622]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:58 smithi196 ceph-mon[76349]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:03:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:03:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:00 smithi184 ceph-mon[100622]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:00 smithi184 ceph-mon[102844]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:00 smithi196 ceph-mon[76349]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:00.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:00.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:00.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:04:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:02 smithi184 ceph-mon[102844]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:04:02.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:02 smithi184 ceph-mon[100622]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:04:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:02 smithi196 ceph-mon[76349]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:04:03.232 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 3h - - 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 6s ago 3h 91.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 3h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 6s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:04:03.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 6s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 3h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 3h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 6s ago 3h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 6s ago 3h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 3h 84.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 6s ago 3h 65.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (2h) 6s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (2h) 8s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:04:03.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (2h) 6s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:04:04.096 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:04:04.097 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:04:04.373 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:04:04.374 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:04:04.374 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:04:04.374 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:04:04.374 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:04:04.374 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:04:04.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[102844]: from='client.57033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[102844]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:04.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[102844]: from='client.67001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[102844]: from='client.67007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/515211993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:04.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[100622]: from='client.57033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[100622]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:04.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[100622]: from='client.67001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[100622]: from='client.67007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/515211993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:04 smithi196 ceph-mon[76349]: from='client.57033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:04 smithi196 ceph-mon[76349]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:04 smithi196 ceph-mon[76349]: from='client.67001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:04 smithi196 ceph-mon[76349]: from='client.67007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/515211993' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:04.684 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:04:04.684 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:04.684 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:04.684 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:04.684 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:05 smithi196 ceph-mon[76349]: from='client.67019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3966431239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[100622]: from='client.67019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3966431239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:05.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[102844]: from='client.67019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3966431239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:06 smithi196 ceph-mon[76349]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:04:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:06 smithi184 ceph-mon[100622]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:04:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:06 smithi184 ceph-mon[102844]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:04:08.456 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:08 smithi196 ceph-mon[76349]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:04:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:08 smithi184 ceph-mon[100622]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:04:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:08 smithi184 ceph-mon[102844]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:04:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:10 smithi196 ceph-mon[76349]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:04:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:10 smithi184 ceph-mon[100622]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:04:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:10 smithi184 ceph-mon[102844]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:04:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:10.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:10.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:10.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:04:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:12 smithi196 ceph-mon[76349]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T23:04:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:12 smithi184 ceph-mon[100622]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T23:04:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:12 smithi184 ceph-mon[102844]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T23:04:14.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:14 smithi184 ceph-mon[100622]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:04:14.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:14.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:14 smithi184 ceph-mon[102844]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:04:14.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:14 smithi196 ceph-mon[76349]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:04:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:15.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:15.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:15.733 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[100622]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:04:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[102844]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:16 smithi196 ceph-mon[76349]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:04:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[102844]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[100622]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:18 smithi196 ceph-mon[76349]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-20T23:04:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:20 smithi184 ceph-mon[100622]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:04:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:20 smithi184 ceph-mon[102844]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:04:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:20 smithi196 ceph-mon[76349]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:04:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:20.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:20.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:20.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:04:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:22 smithi184 ceph-mon[100622]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:04:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:22 smithi184 ceph-mon[102844]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:04:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:22 smithi196 ceph-mon[76349]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:04:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:24 smithi184 ceph-mon[100622]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T23:04:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:24 smithi184 ceph-mon[102844]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T23:04:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:24 smithi196 ceph-mon[76349]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T23:04:25.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:26 smithi184 ceph-mon[100622]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-20T23:04:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:26 smithi184 ceph-mon[102844]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-20T23:04:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:26 smithi196 ceph-mon[76349]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-20T23:04:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:28 smithi184 ceph-mon[100622]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T23:04:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:28 smithi184 ceph-mon[102844]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T23:04:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:28 smithi196 ceph-mon[76349]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T23:04:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:30.512 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:30 smithi196 ceph-mon[76349]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T23:04:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:30 smithi184 ceph-mon[100622]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T23:04:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:30 smithi184 ceph-mon[102844]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T23:04:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:30.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:30.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:30.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:04:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:32 smithi184 ceph-mon[100622]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:04:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:32 smithi184 ceph-mon[102844]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:04:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:32 smithi196 ceph-mon[76349]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:04:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:34 smithi184 ceph-mon[100622]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:04:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:34 smithi184 ceph-mon[102844]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:04:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:34 smithi196 ceph-mon[76349]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:04:34.987 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:04:35.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:35 smithi184 ceph-mon[100622]: from='client.57069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:35.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:35.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:35 smithi184 ceph-mon[102844]: from='client.57069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:35.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 3h - - 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 3h - - 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 19s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 18s ago 3h 91.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:04:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 3h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 19s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 18s ago 3h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 3h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 3h 71.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 3h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 3h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 3h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 3h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 3h 65.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:04:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:04:35.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:04:35.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:04:35.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 3h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:35.869 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:04:35.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:35 smithi196 ceph-mon[76349]: from='client.57069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:04:36.159 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:04:36.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:04:36.479 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:04:36.479 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:36.479 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:36.479 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:36.479 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:04:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='client.67043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1753049130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4126189449' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:04:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='client.67043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1753049130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4126189449' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:04:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='client.67043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:37.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:37.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1753049130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:37.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4126189449' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:37 smithi196 ceph-mon[76349]: from='client.57093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[100622]: from='client.57093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[102844]: from='client.57093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:38 smithi196 ceph-mon[76349]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T23:04:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:38 smithi184 ceph-mon[100622]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T23:04:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:38 smithi184 ceph-mon[102844]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T23:04:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:40 smithi196 ceph-mon[76349]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T23:04:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:40 smithi184 ceph-mon[100622]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T23:04:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:40 smithi184 ceph-mon[102844]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-20T23:04:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:40.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:40.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:40.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:04:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:42 smithi196 ceph-mon[76349]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T23:04:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:42 smithi184 ceph-mon[100622]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T23:04:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:42 smithi184 ceph-mon[102844]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T23:04:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:44 smithi196 ceph-mon[76349]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T23:04:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:44 smithi184 ceph-mon[102844]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T23:04:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:44 smithi184 ceph-mon[100622]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-20T23:04:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:46 smithi196 ceph-mon[76349]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:04:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:46 smithi184 ceph-mon[100622]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:04:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:46 smithi184 ceph-mon[102844]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:04:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:48 smithi196 ceph-mon[76349]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:48 smithi184 ceph-mon[100622]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:48 smithi184 ceph-mon[102844]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:50 smithi196 ceph-mon[76349]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:50 smithi184 ceph-mon[100622]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:50 smithi184 ceph-mon[102844]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:50.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:50.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:04:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:04:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:04:50.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:04:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:04:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:52 smithi196 ceph-mon[76349]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:52 smithi184 ceph-mon[100622]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:52 smithi184 ceph-mon[102844]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:54 smithi184 ceph-mon[100622]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:54 smithi184 ceph-mon[102844]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:54 smithi196 ceph-mon[76349]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:56.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:56.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:56 smithi184 ceph-mon[100622]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:57 smithi184 ceph-mon[102844]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:56 smithi196 ceph-mon[76349]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[100622]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[102844]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:58 smithi196 ceph-mon[76349]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:04:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:00.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:00 smithi184 ceph-mon[100622]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:00.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:00 smithi184 ceph-mon[102844]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:00 smithi196 ceph-mon[76349]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:00.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:00.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:00.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:05:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:02 smithi184 ceph-mon[100622]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:02 smithi184 ceph-mon[102844]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:02 smithi196 ceph-mon[76349]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:05.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:04 smithi184 ceph-mon[100622]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:05.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:04 smithi184 ceph-mon[102844]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:04 smithi196 ceph-mon[76349]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:06.776 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:05:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:06 smithi184 ceph-mon[100622]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:06 smithi184 ceph-mon[102844]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:06 smithi196 ceph-mon[76349]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 3h - - 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 11s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 3h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:05:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 11s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 3h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 3h 65.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:05:07.301 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:05:07.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:05:07.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:05:07.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:05:07.613 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:05:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:05:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:05:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:05:07.615 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:05:07.888 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:05:07.889 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:05:07.889 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:05:07.889 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:05:07.889 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:05:07.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[100622]: from='client.57105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:07.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[100622]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:07.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1510865471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:07.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[102844]: from='client.57105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:07.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[102844]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:07.890 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1510865471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:07 smithi196 ceph-mon[76349]: from='client.57105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:07 smithi196 ceph-mon[76349]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1510865471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:08.199 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:05:08.199 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:08.200 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:08.200 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:08.200 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[100622]: from='client.67079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[100622]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[100622]: from='client.57129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2934917665' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[102844]: from='client.67079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[102844]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[102844]: from='client.57129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2934917665' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:08 smithi196 ceph-mon[76349]: from='client.67079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:08 smithi196 ceph-mon[76349]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:08 smithi196 ceph-mon[76349]: from='client.57129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2934917665' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:10 smithi196 ceph-mon[76349]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:10 smithi184 ceph-mon[100622]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:10 smithi184 ceph-mon[102844]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:10.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:10.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:10.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:05:13.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:12 smithi196 ceph-mon[76349]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:12 smithi184 ceph-mon[100622]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:12 smithi184 ceph-mon[102844]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:15.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:14 smithi184 ceph-mon[100622]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:15.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:14 smithi184 ceph-mon[102844]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:15.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:14 smithi196 ceph-mon[76349]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[100622]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[102844]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:16 smithi196 ceph-mon[76349]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:18 smithi196 ceph-mon[76349]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[100622]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[102844]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:20 smithi196 ceph-mon[76349]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:20 smithi184 ceph-mon[100622]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:20 smithi184 ceph-mon[102844]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:20.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:20.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:20.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:05:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:22 smithi196 ceph-mon[76349]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:22 smithi184 ceph-mon[100622]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:22 smithi184 ceph-mon[102844]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:24 smithi196 ceph-mon[76349]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:24 smithi184 ceph-mon[100622]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:24 smithi184 ceph-mon[102844]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:26 smithi196 ceph-mon[76349]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:26 smithi184 ceph-mon[100622]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:26 smithi184 ceph-mon[102844]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:28 smithi196 ceph-mon[76349]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:28 smithi184 ceph-mon[100622]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:28 smithi184 ceph-mon[102844]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:30 smithi196 ceph-mon[76349]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:30 smithi184 ceph-mon[100622]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:30 smithi184 ceph-mon[102844]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:30.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:30.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:30.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:05:33.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:32 smithi196 ceph-mon[76349]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:32 smithi184 ceph-mon[100622]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:32 smithi184 ceph-mon[102844]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:34 smithi196 ceph-mon[76349]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:34 smithi184 ceph-mon[100622]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:34 smithi184 ceph-mon[102844]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:36.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:36.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[100622]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[102844]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:36 smithi196 ceph-mon[76349]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:38.492 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:05:39.025 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:05:39.025 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T23:05:39.025 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 3h - - 2024-09-20T23:05:39.025 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 2s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 1s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 3h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 2s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 1s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:05:39.026 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 3h 65.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 2s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:05:39.027 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:05:39.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[100622]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:39.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[102844]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:05:39.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:05:39.337 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:05:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:38 smithi196 ceph-mon[76349]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:39.606 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:05:39.607 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:05:39.917 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:05:39.917 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:39.917 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:39.917 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:39.917 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[102844]: from='client.67103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[102844]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[102844]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/607684556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4139653580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[100622]: from='client.67103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[100622]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[100622]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/607684556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4139653580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:39 smithi196 ceph-mon[76349]: from='client.67103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:39 smithi196 ceph-mon[76349]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:39 smithi196 ceph-mon[76349]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/607684556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4139653580' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-mon[102844]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-mon[102844]: from='client.57165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-mon[100622]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-mon[100622]: from='client.57165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:40.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:40.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:40.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:40 smithi196 ceph-mon[76349]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:40 smithi196 ceph-mon[76349]: from='client.57165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:05:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:42 smithi184 ceph-mon[102844]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:42 smithi184 ceph-mon[100622]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:42 smithi196 ceph-mon[76349]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:44.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:44 smithi196 ceph-mon[76349]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:44 smithi184 ceph-mon[100622]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:44 smithi184 ceph-mon[102844]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:46 smithi184 ceph-mon[102844]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:46 smithi184 ceph-mon[100622]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:46 smithi196 ceph-mon[76349]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:48 smithi184 ceph-mon[100622]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:48 smithi184 ceph-mon[102844]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:48 smithi196 ceph-mon[76349]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:51.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:50 smithi196 ceph-mon[76349]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:50 smithi184 ceph-mon[100622]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:50 smithi184 ceph-mon[102844]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:50.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:50.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:05:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:05:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:05:50.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:05:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:05:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:52 smithi184 ceph-mon[100622]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:52 smithi184 ceph-mon[102844]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:52 smithi196 ceph-mon[76349]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:54 smithi184 ceph-mon[102844]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:54 smithi184 ceph-mon[100622]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:54 smithi196 ceph-mon[76349]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[102844]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[100622]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:56 smithi196 ceph-mon[76349]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[100622]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[102844]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:59 smithi196 ceph-mon[76349]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:05:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:01 smithi184 ceph-mon[100622]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:01 smithi184 ceph-mon[102844]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:00.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:00.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:00.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:01 smithi196 ceph-mon[76349]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:06:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:03 smithi196 ceph-mon[76349]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:03.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:03 smithi184 ceph-mon[100622]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:03.448 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:03 smithi184 ceph-mon[102844]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:05 smithi196 ceph-mon[76349]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:05 smithi184 ceph-mon[100622]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:05 smithi184 ceph-mon[102844]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:07 smithi196 ceph-mon[76349]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:07 smithi184 ceph-mon[100622]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:07 smithi184 ceph-mon[102844]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:09 smithi196 ceph-mon[76349]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:09 smithi184 ceph-mon[100622]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:09 smithi184 ceph-mon[102844]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:10.212 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 3h - - 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 3h 350M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:06:10.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:06:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 3h 65.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:06:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:06:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:06:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:06:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:06:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:10.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:10.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:11.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:10.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:11.049 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:06:11.049 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:06:11.049 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:06:11.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:11.049 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:06:11.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:06:11.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:06:11.317 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:06:11.318 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:06:11.318 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:11 smithi184 ceph-mon[100622]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:11.318 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4247919296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:11.319 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:11 smithi184 ceph-mon[102844]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:11.319 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4247919296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:11 smithi196 ceph-mon[76349]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4247919296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:11.626 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:06:11.626 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:11.626 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[102844]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[102844]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[102844]: from='client.67151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3997425172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[100622]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[100622]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[100622]: from='client.67151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3997425172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:12 smithi196 ceph-mon[76349]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:12 smithi196 ceph-mon[76349]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:12 smithi196 ceph-mon[76349]: from='client.67151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3997425172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:13 smithi196 ceph-mon[76349]: from='client.67163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:13 smithi196 ceph-mon[76349]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:13 smithi184 ceph-mon[100622]: from='client.67163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:13 smithi184 ceph-mon[100622]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:13 smithi184 ceph-mon[102844]: from='client.67163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:13 smithi184 ceph-mon[102844]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:14.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:15 smithi196 ceph-mon[76349]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:15 smithi184 ceph-mon[100622]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:15 smithi184 ceph-mon[102844]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:16.536 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:16.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:17 smithi196 ceph-mon[76349]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[100622]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[102844]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:17.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[100622]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:19.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.304 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[102844]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:19.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:19 smithi196 ceph-mon[76349]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:21 smithi184 ceph-mon[100622]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:21 smithi184 ceph-mon[102844]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:20.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:20.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:20.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:21 smithi196 ceph-mon[76349]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:06:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:23 smithi196 ceph-mon[76349]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:23 smithi184 ceph-mon[102844]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:23 smithi184 ceph-mon[100622]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:25 smithi196 ceph-mon[76349]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:25 smithi184 ceph-mon[102844]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:25 smithi184 ceph-mon[100622]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:27 smithi196 ceph-mon[76349]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:27 smithi184 ceph-mon[102844]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:27 smithi184 ceph-mon[100622]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:28 smithi196 ceph-mon[76349]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:28 smithi184 ceph-mon[100622]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:28 smithi184 ceph-mon[102844]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:30.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:30.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:30.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:30 smithi184 ceph-mon[100622]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:30.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:30 smithi184 ceph-mon[102844]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:30 smithi196 ceph-mon[76349]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:30.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:30.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:30.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:06:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:32 smithi184 ceph-mon[100622]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:32 smithi184 ceph-mon[102844]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:32 smithi196 ceph-mon[76349]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:34 smithi184 ceph-mon[100622]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:34 smithi184 ceph-mon[102844]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:34 smithi196 ceph-mon[76349]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:36 smithi184 ceph-mon[100622]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:36 smithi184 ceph-mon[102844]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:36 smithi196 ceph-mon[76349]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[100622]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.801 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.802 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[102844]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:38.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:38.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:38 smithi196 ceph-mon[76349]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:40 smithi184 ceph-mon[100622]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:40 smithi184 ceph-mon[102844]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:40 smithi196 ceph-mon[76349]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:40.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:40.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:40.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:41.927 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:06:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T23:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 3h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 3h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 3h 67.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:06:42.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:42 smithi184 ceph-mon[100622]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:42.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:42 smithi184 ceph-mon[100622]: from='client.67175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:42.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:42 smithi184 ceph-mon[102844]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:42.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:42 smithi184 ceph-mon[102844]: from='client.67175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:42.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:06:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:06:42.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:06:43.036 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:06:43.037 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:06:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:42 smithi196 ceph-mon[76349]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:42 smithi196 ceph-mon[76349]: from='client.67175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:43.351 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:06:43.351 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:43.351 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:44.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:43 smithi196 ceph-mon[76349]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:43 smithi196 ceph-mon[76349]: from='client.57225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1761432730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:44.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:43 smithi196 ceph-mon[76349]: from='client.67199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3846297324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[100622]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[100622]: from='client.57225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1761432730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[100622]: from='client.67199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3846297324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[102844]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[102844]: from='client.57225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1761432730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[102844]: from='client.67199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3846297324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:44 smithi184 ceph-mon[100622]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:44 smithi184 ceph-mon[102844]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:44 smithi196 ceph-mon[76349]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:46 smithi184 ceph-mon[100622]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:46 smithi184 ceph-mon[102844]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:46 smithi196 ceph-mon[76349]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:48 smithi184 ceph-mon[100622]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:48 smithi184 ceph-mon[102844]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:48 smithi196 ceph-mon[76349]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:50.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:50 smithi184 ceph-mon[100622]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:50.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:50 smithi184 ceph-mon[102844]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:50 smithi196 ceph-mon[76349]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:50.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:50.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:06:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:06:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:06:50.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:06:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:06:52.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:52 smithi196 ceph-mon[76349]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:52 smithi184 ceph-mon[100622]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:52 smithi184 ceph-mon[102844]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:54 smithi196 ceph-mon[76349]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:54 smithi184 ceph-mon[100622]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:54 smithi184 ceph-mon[102844]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:56 smithi184 ceph-mon[100622]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:56 smithi184 ceph-mon[102844]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:56 smithi196 ceph-mon[76349]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[100622]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[102844]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:58 smithi196 ceph-mon[76349]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:06:59.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:00.160 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:00.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:01.074 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:00 smithi184 ceph-mon[100622]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:01.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:00 smithi184 ceph-mon[102844]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:01.074 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:00.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:01.074 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:00.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:01.074 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:00.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:00 smithi196 ceph-mon[76349]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:07:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:02 smithi196 ceph-mon[76349]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:02 smithi184 ceph-mon[100622]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:02 smithi184 ceph-mon[102844]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:05.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:04 smithi184 ceph-mon[100622]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:05.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:04 smithi184 ceph-mon[102844]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:04 smithi196 ceph-mon[76349]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:06 smithi196 ceph-mon[76349]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:06 smithi184 ceph-mon[100622]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:06 smithi184 ceph-mon[102844]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:08 smithi196 ceph-mon[76349]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:08 smithi184 ceph-mon[100622]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:08 smithi184 ceph-mon[102844]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:10 smithi196 ceph-mon[76349]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:10 smithi184 ceph-mon[100622]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:10 smithi184 ceph-mon[102844]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:10.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:10.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:10.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:07:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:12 smithi196 ceph-mon[76349]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:12 smithi184 ceph-mon[100622]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:12 smithi184 ceph-mon[102844]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:13.648 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:07:14.176 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 3h - - 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 15s ago 3h 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 17s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 15s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 17s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 17s ago 3h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 15s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 17s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 15s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:07:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 17s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 17s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 17s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 17s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 15s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 15s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 15s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 15s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 15s ago 3h 67.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 15s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 17s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:07:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 15s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:07:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:07:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:07:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:07:14.484 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:07:14.751 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:07:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:07:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:07:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:07:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:07:14.752 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:07:14.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:14 smithi196 ceph-mon[76349]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:14.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:14 smithi196 ceph-mon[76349]: from='client.67211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:14.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:14.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:14 smithi196 ceph-mon[76349]: from='client.67217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:14.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3992037675' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[100622]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[100622]: from='client.67211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[100622]: from='client.67217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3992037675' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[102844]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[102844]: from='client.67211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[102844]: from='client.67217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3992037675' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:15.063 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:07:15.063 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:15.063 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:15.063 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:15.063 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:15 smithi196 ceph-mon[76349]: from='client.67223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:15 smithi196 ceph-mon[76349]: from='client.57273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/132678061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[100622]: from='client.67223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[100622]: from='client.57273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/132678061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[102844]: from='client.67223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[102844]: from='client.57273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/132678061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:16 smithi196 ceph-mon[76349]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:16 smithi184 ceph-mon[100622]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:16 smithi184 ceph-mon[102844]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[100622]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[102844]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:18 smithi196 ceph-mon[76349]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:20 smithi184 ceph-mon[100622]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:20 smithi184 ceph-mon[102844]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:20.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:20.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:20.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:20 smithi196 ceph-mon[76349]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:07:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:22 smithi184 ceph-mon[100622]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:22 smithi184 ceph-mon[102844]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:22 smithi196 ceph-mon[76349]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:24 smithi184 ceph-mon[100622]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:24 smithi184 ceph-mon[102844]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:24 smithi196 ceph-mon[76349]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:26 smithi184 ceph-mon[100622]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:26 smithi184 ceph-mon[102844]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:26 smithi196 ceph-mon[76349]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:28 smithi184 ceph-mon[100622]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:28 smithi184 ceph-mon[102844]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:28 smithi196 ceph-mon[76349]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:30 smithi184 ceph-mon[100622]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:30 smithi184 ceph-mon[102844]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:30.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:30.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:30.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:30 smithi196 ceph-mon[76349]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:07:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:32 smithi184 ceph-mon[100622]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:32 smithi184 ceph-mon[102844]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:32 smithi196 ceph-mon[76349]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:35.226 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:34 smithi196 ceph-mon[76349]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:34 smithi184 ceph-mon[100622]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:34 smithi184 ceph-mon[102844]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:36 smithi184 ceph-mon[100622]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:36 smithi184 ceph-mon[102844]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:36 smithi196 ceph-mon[76349]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:39 smithi196 ceph-mon[76349]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[100622]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[102844]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:41.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:41 smithi184 ceph-mon[100622]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:41.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:41 smithi184 ceph-mon[102844]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:41.286 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:40.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:41.286 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:40.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:41.287 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:40.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:41 smithi196 ceph-mon[76349]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:07:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:43 smithi196 ceph-mon[76349]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:43 smithi184 ceph-mon[100622]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:43 smithi184 ceph-mon[102844]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:45.363 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:45 smithi184 ceph-mon[100622]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:45.363 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:45 smithi184 ceph-mon[102844]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:07:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:45 smithi196 ceph-mon[76349]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 3h - - 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 3h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:07:45.891 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 8s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 3h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 8s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:07:45.892 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:07:46.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:07:46.202 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:07:46.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:07:46.472 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:07:46.781 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:07:46.781 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:46.781 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:46.781 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:46.781 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.57285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/215112344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2695185568' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.57285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/215112344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2695185568' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.57285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/215112344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:47.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2695185568' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:48 smithi196 ceph-mon[76349]: from='client.67271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:48 smithi184 ceph-mon[100622]: from='client.67271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:48 smithi184 ceph-mon[102844]: from='client.67271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:49 smithi196 ceph-mon[76349]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:49 smithi184 ceph-mon[100622]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:49 smithi184 ceph-mon[102844]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:51 smithi184 ceph-mon[100622]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:51 smithi184 ceph-mon[102844]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:50.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:50.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:07:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:07:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:07:50.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:07:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:51 smithi196 ceph-mon[76349]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:07:52] "GET /metrics HTTP/1.1" 200 38927 "" "Prometheus/2.43.0" 2024-09-20T23:07:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:53 smithi196 ceph-mon[76349]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:53 smithi184 ceph-mon[100622]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:53 smithi184 ceph-mon[102844]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:55 smithi196 ceph-mon[76349]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:55 smithi184 ceph-mon[100622]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:55 smithi184 ceph-mon[102844]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:57.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:57.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:57 smithi196 ceph-mon[76349]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:57 smithi184 ceph-mon[100622]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:57 smithi184 ceph-mon[102844]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[100622]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:58 smithi184 ceph-mon[102844]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:07:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:58 smithi196 ceph-mon[76349]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:00 smithi196 ceph-mon[76349]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:01.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:00 smithi184 ceph-mon[100622]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:01.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:00 smithi184 ceph-mon[102844]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:01.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:00.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:01.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:00.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:01.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:00.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:02.243 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:08:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:02 smithi184 ceph-mon[100622]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:02 smithi184 ceph-mon[102844]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:02 smithi196 ceph-mon[76349]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:04 smithi196 ceph-mon[76349]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:05.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:04 smithi184 ceph-mon[100622]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:04 smithi184 ceph-mon[102844]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:08:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:06 smithi196 ceph-mon[76349]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:06 smithi184 ceph-mon[100622]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:06 smithi184 ceph-mon[102844]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:08 smithi196 ceph-mon[76349]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:08 smithi184 ceph-mon[100622]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:08 smithi184 ceph-mon[102844]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:10 smithi196 ceph-mon[76349]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:10 smithi184 ceph-mon[102844]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:10 smithi184 ceph-mon[100622]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:10.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:10.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:10.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:08:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:12 smithi196 ceph-mon[76349]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:12 smithi184 ceph-mon[100622]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:12 smithi184 ceph-mon[102844]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:08:13.810650384Z level=info msg="Completed cleanup jobs" duration=11.136399ms 2024-09-20T23:08:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:14 smithi196 ceph-mon[76349]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:15.197 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:14 smithi184 ceph-mon[100622]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:15.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:15.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:14 smithi184 ceph-mon[102844]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:15.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:17.078 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:08:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:16 smithi196 ceph-mon[76349]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:16 smithi184 ceph-mon[100622]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:16 smithi184 ceph-mon[102844]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:08:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T23:08:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 3h - - 2024-09-20T23:08:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:08:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 18s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 3h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 18s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:08:17.619 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:08:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 3h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:08:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:08:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:08:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:08:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:08:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:08:17.935 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:08:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:17 smithi196 ceph-mon[76349]: from='client.57321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.204 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:08:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:08:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:08:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:08:18.205 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[100622]: from='client.57321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[102844]: from='client.57321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:18.515 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:08:18.515 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:18.515 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:18.515 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:18.515 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:18 smithi196 ceph-mon[76349]: from='client.57327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:18 smithi196 ceph-mon[76349]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:18 smithi196 ceph-mon[76349]: from='client.57333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1190213421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4196946153' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[100622]: from='client.57327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[100622]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[100622]: from='client.57333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1190213421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4196946153' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[102844]: from='client.57327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[102844]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[102844]: from='client.57333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1190213421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4196946153' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[102844]: from='client.57345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[100622]: from='client.57345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:19 smithi196 ceph-mon[76349]: from='client.57345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:20 smithi184 ceph-mon[100622]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:20 smithi184 ceph-mon[102844]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:20.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:20.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:20.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:20 smithi196 ceph-mon[76349]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:08:23.034 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:22 smithi184 ceph-mon[100622]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.034 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:22 smithi184 ceph-mon[102844]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:22 smithi196 ceph-mon[76349]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:24 smithi196 ceph-mon[76349]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:24 smithi184 ceph-mon[100622]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:24 smithi184 ceph-mon[102844]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:26 smithi184 ceph-mon[100622]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:26 smithi184 ceph-mon[102844]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:26 smithi196 ceph-mon[76349]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:28 smithi184 ceph-mon[100622]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:29.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:28 smithi184 ceph-mon[102844]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:28 smithi196 ceph-mon[76349]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:30 smithi184 ceph-mon[100622]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:30 smithi184 ceph-mon[102844]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:30.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:30.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:31.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:30.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:30 smithi196 ceph-mon[76349]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:08:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:32 smithi184 ceph-mon[100622]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:32 smithi184 ceph-mon[102844]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:32 smithi196 ceph-mon[76349]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:34 smithi184 ceph-mon[100622]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:34 smithi184 ceph-mon[102844]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:34 smithi196 ceph-mon[76349]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:37.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:37.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:36 smithi196 ceph-mon[76349]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:36 smithi184 ceph-mon[100622]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:36 smithi184 ceph-mon[102844]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:38 smithi196 ceph-mon[76349]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[100622]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[102844]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:41 smithi184 ceph-mon[100622]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:41 smithi184 ceph-mon[102844]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:40.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:40.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:40.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:41 smithi196 ceph-mon[76349]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:08:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:43 smithi196 ceph-mon[76349]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:43 smithi184 ceph-mon[102844]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.466 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:43 smithi184 ceph-mon[100622]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:45.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:45 smithi184 ceph-mon[100622]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:45.230 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:45 smithi184 ceph-mon[102844]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:45 smithi196 ceph-mon[76349]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:47 smithi196 ceph-mon[76349]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:47 smithi184 ceph-mon[100622]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:47 smithi184 ceph-mon[102844]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:48.812 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:08:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:08:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 3h - - 2024-09-20T23:08:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 3h - - 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 11s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 10s ago 3h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 10s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 3h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 10s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 11s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 10s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:08:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 10s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 10s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 10s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 3h 70.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 10s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:08:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 10s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:08:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:49 smithi196 ceph-mon[76349]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:49 smithi184 ceph-mon[102844]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:49.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:49 smithi184 ceph-mon[100622]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:49.649 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:08:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:08:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:08:49.651 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:08:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:08:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:08:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:08:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:08:49.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:08:50.230 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:08:50.230 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:50.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:50.231 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:50.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:50.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[100622]: from='client.67319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[100622]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1433328588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:50.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[102844]: from='client.67319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[102844]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1433328588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:50 smithi196 ceph-mon[76349]: from='client.67319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:50 smithi196 ceph-mon[76349]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1433328588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[102844]: from='client.57369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[102844]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[102844]: from='client.67343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/819489171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[100622]: from='client.57369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[100622]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[100622]: from='client.67343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/819489171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:50.971Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:50.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:08:51.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:08:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:08:50.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:51 smithi196 ceph-mon[76349]: from='client.57369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:51 smithi196 ceph-mon[76349]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:51 smithi196 ceph-mon[76349]: from='client.67343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/819489171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:08:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:08:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:53 smithi196 ceph-mon[76349]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:53 smithi184 ceph-mon[100622]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:53 smithi184 ceph-mon[102844]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:55.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:55 smithi196 ceph-mon[76349]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:55 smithi184 ceph-mon[100622]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:55 smithi184 ceph-mon[102844]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:57 smithi196 ceph-mon[76349]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:57 smithi184 ceph-mon[100622]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:57 smithi184 ceph-mon[102844]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:58 smithi196 ceph-mon[76349]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[100622]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[102844]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:08:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:00 smithi196 ceph-mon[76349]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:00.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:00 smithi184 ceph-mon[100622]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:00.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:00 smithi184 ceph-mon[102844]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:00.971Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:00.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:00.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:09:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:02 smithi184 ceph-mon[102844]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:02 smithi184 ceph-mon[100622]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:02 smithi196 ceph-mon[76349]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:04 smithi184 ceph-mon[100622]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:04 smithi184 ceph-mon[102844]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:04 smithi196 ceph-mon[76349]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:06 smithi196 ceph-mon[76349]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:06 smithi184 ceph-mon[100622]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:07.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:06 smithi184 ceph-mon[102844]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:08 smithi196 ceph-mon[76349]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:08 smithi184 ceph-mon[100622]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:08 smithi184 ceph-mon[102844]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:10 smithi196 ceph-mon[76349]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:10.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:10 smithi184 ceph-mon[100622]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:10.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:10 smithi184 ceph-mon[102844]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:10.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:10.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:10.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:09:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:12 smithi196 ceph-mon[76349]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:12 smithi184 ceph-mon[100622]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:12 smithi184 ceph-mon[102844]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:14 smithi196 ceph-mon[76349]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:14 smithi184 ceph-mon[102844]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:14 smithi184 ceph-mon[100622]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:16 smithi196 ceph-mon[76349]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:16 smithi184 ceph-mon[100622]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:16 smithi184 ceph-mon[102844]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[100622]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[102844]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:18 smithi196 ceph-mon[76349]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:19.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:19.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:09:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:20 smithi196 ceph-mon[76349]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[102844]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-mon[100622]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 3h - - 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 1s ago 3h 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 3s ago 3h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 3h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 3s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:09:21.053 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 1s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 3s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 3s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 3s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 3s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 3h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:09:21.054 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:09:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:20.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:20.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:20.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:21.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:09:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:09:21.390 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:09:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:21 smithi196 ceph-mon[76349]: from='client.57393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:21 smithi196 ceph-mon[76349]: from='client.57399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:21 smithi196 ceph-mon[76349]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:09:21.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:09:21.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:09:21.664 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[100622]: from='client.57393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.664 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[100622]: from='client.57399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.664 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[100622]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.664 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[102844]: from='client.57393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.665 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[102844]: from='client.57399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.665 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:21 smithi184 ceph-mon[102844]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:21.974 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:09:21.974 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:21.974 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:21.974 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:21.974 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:09:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:22 smithi196 ceph-mon[76349]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/447105116' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:22 smithi196 ceph-mon[76349]: from='client.57417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3203157028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[100622]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/447105116' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[100622]: from='client.57417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3203157028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[102844]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/447105116' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[102844]: from='client.57417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:22.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3203157028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:24 smithi184 ceph-mon[100622]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:24 smithi184 ceph-mon[102844]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:24 smithi196 ceph-mon[76349]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:26 smithi184 ceph-mon[100622]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:26 smithi184 ceph-mon[102844]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:26 smithi196 ceph-mon[76349]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:28 smithi184 ceph-mon[100622]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:28 smithi184 ceph-mon[102844]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:28 smithi196 ceph-mon[76349]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:29.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:30 smithi184 ceph-mon[100622]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:30 smithi184 ceph-mon[102844]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:30 smithi196 ceph-mon[76349]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:30.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:30.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:30.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:32] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T23:09:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:32 smithi184 ceph-mon[100622]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:32 smithi184 ceph-mon[102844]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:32 smithi196 ceph-mon[76349]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:34 smithi184 ceph-mon[100622]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:34 smithi184 ceph-mon[102844]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:34 smithi196 ceph-mon[76349]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:36 smithi184 ceph-mon[100622]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:36 smithi184 ceph-mon[102844]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:36 smithi196 ceph-mon[76349]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:39.098 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:38 smithi196 ceph-mon[76349]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:39.098 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.098 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.098 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[100622]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[102844]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[100622]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[102844]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:40 smithi196 ceph-mon[76349]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:40.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:40.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:40.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:42] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T23:09:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:42 smithi184 ceph-mon[100622]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:42 smithi184 ceph-mon[102844]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:42 smithi196 ceph-mon[76349]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:44.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:44 smithi184 ceph-mon[100622]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:44.840 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:44.840 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:44 smithi184 ceph-mon[102844]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:44.840 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:44 smithi196 ceph-mon[76349]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:46.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:46.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:46 smithi184 ceph-mon[100622]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:46.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:46.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:46 smithi184 ceph-mon[102844]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:46 smithi196 ceph-mon[76349]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:48 smithi184 ceph-mon[102844]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:49.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:48 smithi184 ceph-mon[100622]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:48 smithi196 ceph-mon[76349]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:50.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:50 smithi184 ceph-mon[102844]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:50.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:50 smithi184 ceph-mon[100622]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:50 smithi196 ceph-mon[76349]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:50.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:50.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:09:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:09:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:09:50.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:09:52.269 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:09:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:09:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 3h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:09:52.803 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 13s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 18.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 13s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 13s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 13s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 3h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:09:52.804 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:09:52.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 13s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:09:52.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:09:52.805 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 13s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:09:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:52 smithi184 ceph-mon[100622]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:52 smithi184 ceph-mon[102844]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:53.111 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:09:53.111 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:09:53.111 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:09:53.112 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:09:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:52 smithi196 ceph-mon[76349]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:53.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:09:53.381 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:09:53.382 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:09:53.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:09:53.692 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:09:53.692 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:53.692 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:53.692 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:53.693 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[100622]: from='client.67391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[100622]: from='client.67397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[100622]: from='client.67403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/361535437' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[102844]: from='client.67391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[102844]: from='client.67397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[102844]: from='client.67403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/361535437' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:53 smithi196 ceph-mon[76349]: from='client.67391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:53 smithi196 ceph-mon[76349]: from='client.67397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:53 smithi196 ceph-mon[76349]: from='client.67403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/361535437' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[100622]: from='client.67415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[100622]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1534259738' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[102844]: from='client.67415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[102844]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1534259738' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:54 smithi196 ceph-mon[76349]: from='client.67415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:54 smithi196 ceph-mon[76349]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1534259738' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:57.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:57.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:56 smithi184 ceph-mon[102844]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:57.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:56 smithi184 ceph-mon[100622]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:56 smithi196 ceph-mon[76349]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:59.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:58 smithi196 ceph-mon[76349]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:59.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:09:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[102844]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[100622]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:09:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:09:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:00 smithi196 ceph-mon[76349]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[102844]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[100622]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:00.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:00.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:00.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:00.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:02 smithi196 ceph-mon[76349]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:02 smithi184 ceph-mon[102844]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:02 smithi184 ceph-mon[100622]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:04 smithi196 ceph-mon[76349]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:04 smithi184 ceph-mon[100622]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:05.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:04 smithi184 ceph-mon[102844]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:06.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:06.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:06 smithi184 ceph-mon[100622]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:06.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:06.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:06 smithi184 ceph-mon[102844]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:06 smithi196 ceph-mon[76349]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:08 smithi196 ceph-mon[76349]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:08 smithi184 ceph-mon[100622]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:08 smithi184 ceph-mon[102844]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:10 smithi196 ceph-mon[76349]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:10.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:10 smithi184 ceph-mon[100622]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:10.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:10 smithi184 ceph-mon[102844]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:10.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:10.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:10.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:12 smithi196 ceph-mon[76349]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:13.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:12 smithi184 ceph-mon[100622]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:12 smithi184 ceph-mon[102844]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:14 smithi196 ceph-mon[76349]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:14 smithi184 ceph-mon[100622]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:14 smithi184 ceph-mon[102844]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:16 smithi196 ceph-mon[76349]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:16 smithi184 ceph-mon[100622]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:16 smithi184 ceph-mon[102844]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:19.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:19 smithi196 ceph-mon[76349]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:19.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[100622]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[102844]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:20 smithi196 ceph-mon[76349]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:20.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[102844]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[100622]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:20.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:20.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:20.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:20.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:22 smithi184 ceph-mon[100622]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:22 smithi184 ceph-mon[102844]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:22 smithi196 ceph-mon[76349]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 3h - - 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 6s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 6s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 6s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 6s ago 3h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:10:24.509 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 6s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 6s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 6s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 6s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 6s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 6s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 3h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:10:24.510 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:10:24.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 6s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:10:24.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:10:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:24 smithi184 ceph-mon[100622]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:24 smithi184 ceph-mon[100622]: from='client.67427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:24 smithi184 ceph-mon[102844]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:24 smithi184 ceph-mon[102844]: from='client.67427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:24.820 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:10:24.820 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:10:24.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:10:24.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:10:24.821 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:10:25.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:10:25.091 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:10:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:24 smithi196 ceph-mon[76349]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:24 smithi196 ceph-mon[76349]: from='client.67427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.404 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:25.404 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:25.404 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:25.404 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:25.404 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[102844]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[102844]: from='client.67439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4195955805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[102844]: from='client.67451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/744502215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:25.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[100622]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[100622]: from='client.67439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4195955805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:25.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[100622]: from='client.67451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:25.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/744502215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:25 smithi196 ceph-mon[76349]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:25 smithi196 ceph-mon[76349]: from='client.67439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4195955805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:25 smithi196 ceph-mon[76349]: from='client.67451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/744502215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:26 smithi184 ceph-mon[100622]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:26 smithi184 ceph-mon[102844]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:26 smithi196 ceph-mon[76349]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:28 smithi184 ceph-mon[100622]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:28 smithi184 ceph-mon[102844]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:28 smithi196 ceph-mon[76349]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:30.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:30 smithi184 ceph-mon[100622]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:30.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:30 smithi184 ceph-mon[102844]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:30 smithi196 ceph-mon[76349]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:30.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:30.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:30.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:32 smithi184 ceph-mon[100622]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:32 smithi184 ceph-mon[102844]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:32 smithi196 ceph-mon[76349]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:34 smithi184 ceph-mon[100622]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:34 smithi184 ceph-mon[102844]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:34 smithi196 ceph-mon[76349]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:36 smithi184 ceph-mon[100622]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:36 smithi184 ceph-mon[102844]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:36 smithi196 ceph-mon[76349]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[102844]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[100622]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:38 smithi196 ceph-mon[76349]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:40 smithi196 ceph-mon[76349]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[100622]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[102844]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:40.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:40.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:40.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:42 smithi196 ceph-mon[76349]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:42 smithi184 ceph-mon[100622]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:42 smithi184 ceph-mon[102844]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:44 smithi196 ceph-mon[76349]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:44 smithi184 ceph-mon[100622]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:44 smithi184 ceph-mon[102844]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:47.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:47.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:46 smithi184 ceph-mon[100622]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:47.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:47.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:46 smithi184 ceph-mon[102844]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:46 smithi196 ceph-mon[76349]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:48.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:48 smithi184 ceph-mon[100622]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:48.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:48 smithi184 ceph-mon[102844]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:48 smithi196 ceph-mon[76349]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:50 smithi196 ceph-mon[76349]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:50 smithi184 ceph-mon[100622]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:50 smithi184 ceph-mon[102844]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:50.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:50.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:10:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:10:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:10:50.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:10:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:10:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:10:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:52 smithi196 ceph-mon[76349]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:52 smithi184 ceph-mon[100622]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:52 smithi184 ceph-mon[102844]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:54 smithi196 ceph-mon[76349]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:54 smithi184 ceph-mon[100622]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:54 smithi184 ceph-mon[102844]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:55.702 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:10:56.228 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:10:56.228 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T23:10:56.228 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T23:10:56.228 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 18s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 18s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 18s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 3h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 18s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 18s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 18s ago 3h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 18s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 16s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 16s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:10:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 16s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:10:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 16s ago 3h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:10:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:10:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 16s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:10:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:10:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 16s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:10:56.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:10:56.539 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:10:56.808 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:10:56.808 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:10:56.808 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:10:56.809 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:10:57.119 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:57.119 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:57.119 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:57.119 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:57.119 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[100622]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:57.119 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[100622]: from='client.67463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[100622]: from='client.57507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4198923290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[102844]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[102844]: from='client.67463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[102844]: from='client.57507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.120 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4198923290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:56 smithi196 ceph-mon[76349]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:56 smithi196 ceph-mon[76349]: from='client.67463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:56 smithi196 ceph-mon[76349]: from='client.57507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4198923290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:57 smithi196 ceph-mon[76349]: from='client.57513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:57 smithi196 ceph-mon[76349]: from='client.67487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3486982990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[100622]: from='client.57513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[100622]: from='client.67487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3486982990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[102844]: from='client.57513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[102844]: from='client.67487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:58.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3486982990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:58 smithi196 ceph-mon[76349]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[100622]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[102844]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:10:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:10:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:10:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:10:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:01.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:00 smithi196 ceph-mon[76349]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:01.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[100622]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[102844]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:00.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:00.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:00.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:11:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:02 smithi184 ceph-mon[100622]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:02 smithi184 ceph-mon[102844]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:02 smithi196 ceph-mon[76349]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:04 smithi184 ceph-mon[100622]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:04 smithi184 ceph-mon[102844]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:04 smithi196 ceph-mon[76349]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:07.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:07.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:06 smithi184 ceph-mon[100622]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:07.244 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:07.244 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:06 smithi184 ceph-mon[102844]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:06 smithi196 ceph-mon[76349]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:09.244 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:08 smithi184 ceph-mon[100622]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:09.245 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:08 smithi184 ceph-mon[102844]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:08 smithi196 ceph-mon[76349]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:10 smithi184 ceph-mon[100622]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:10 smithi184 ceph-mon[102844]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:10.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:10.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:10.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:10 smithi196 ceph-mon[76349]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:11:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:12 smithi184 ceph-mon[100622]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:12 smithi184 ceph-mon[102844]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:12 smithi196 ceph-mon[76349]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:14.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:14 smithi196 ceph-mon[76349]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:14 smithi184 ceph-mon[102844]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:14 smithi184 ceph-mon[100622]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:16 smithi184 ceph-mon[100622]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:17.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:16 smithi184 ceph-mon[102844]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:16 smithi196 ceph-mon[76349]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[100622]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[102844]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:18 smithi196 ceph-mon[76349]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[100622]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[102844]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:20.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:20.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:20.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:21 smithi196 ceph-mon[76349]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:11:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:23 smithi196 ceph-mon[76349]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:23 smithi184 ceph-mon[100622]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:23 smithi184 ceph-mon[102844]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:25.378 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:25 smithi184 ceph-mon[100622]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:25.378 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:25 smithi184 ceph-mon[102844]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:25 smithi196 ceph-mon[76349]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:27 smithi196 ceph-mon[76349]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:27.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:27.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:27 smithi184 ceph-mon[100622]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:27.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:27.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:27 smithi184 ceph-mon[102844]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:27.418 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:11:27.958 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 9s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 9s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 9s ago 3h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 9s ago 3h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:11:27.959 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 9s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 9s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 9s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 9s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 3h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:11:27.960 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:11:27.961 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:11:28.278 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:11:28.279 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:11:28.549 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:11:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:11:28.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:11:28.883 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:11:28.883 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:28.883 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:28.883 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:29.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='client.57537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:29.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1162622758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/413158846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='client.57537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1162622758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/413158846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:29.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='client.57537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1162622758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/413158846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:30.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:30 smithi184 ceph-mon[100622]: from='client.67523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:30.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:30 smithi184 ceph-mon[102844]: from='client.67523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:30 smithi196 ceph-mon[76349]: from='client.67523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:31 smithi184 ceph-mon[100622]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:31 smithi184 ceph-mon[102844]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:30.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:30.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:30.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:31 smithi196 ceph-mon[76349]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:11:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:33 smithi196 ceph-mon[76349]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.391 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:33 smithi184 ceph-mon[100622]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:33 smithi184 ceph-mon[102844]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:35.389 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:35 smithi184 ceph-mon[100622]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:35.389 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:35 smithi184 ceph-mon[102844]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:35 smithi196 ceph-mon[76349]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:37 smithi196 ceph-mon[76349]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:37 smithi184 ceph-mon[100622]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:37 smithi184 ceph-mon[102844]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:39 smithi196 ceph-mon[76349]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[100622]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[102844]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[100622]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[102844]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:40.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:40.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:40.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:41.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:41 smithi196 ceph-mon[76349]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:41.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:11:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:43 smithi184 ceph-mon[100622]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:43 smithi184 ceph-mon[102844]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:43 smithi196 ceph-mon[76349]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:45 smithi196 ceph-mon[76349]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:45.391 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:45 smithi184 ceph-mon[100622]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:45.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:45 smithi184 ceph-mon[102844]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:47 smithi196 ceph-mon[76349]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:47 smithi184 ceph-mon[102844]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:47 smithi184 ceph-mon[100622]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:49.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:49 smithi184 ceph-mon[100622]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:49.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:49 smithi184 ceph-mon[102844]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:49 smithi196 ceph-mon[76349]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:51 smithi184 ceph-mon[100622]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:51 smithi184 ceph-mon[102844]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:50.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:50.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:11:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:11:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:11:50.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:11:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:51 smithi196 ceph-mon[76349]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:11:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:11:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:53 smithi184 ceph-mon[100622]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:53 smithi184 ceph-mon[102844]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:53 smithi196 ceph-mon[76349]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:55.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:55 smithi184 ceph-mon[100622]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:55.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:55 smithi184 ceph-mon[102844]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:55 smithi196 ceph-mon[76349]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:57 smithi184 ceph-mon[100622]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:57 smithi184 ceph-mon[102844]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:57 smithi196 ceph-mon[76349]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:59.171 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:11:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[100622]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[102844]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:59 smithi196 ceph-mon[76349]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:11:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 3h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 19s ago 3h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:11:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 19s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 19s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 19s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 19s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 19s ago 3h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 19s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:11:59.703 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 19s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:12:00.011 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:12:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:12:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:12:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:12:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:12:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:12:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:12:00.014 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:12:00.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:12:00.284 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:12:00.285 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:12:00.285 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:12:00.285 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:12:00.285 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:12:00.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-mon[100622]: from='client.67535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:00.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/118837105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:00.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-mon[102844]: from='client.67535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:00.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/118837105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:00.600 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:12:00.600 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:00.600 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:00 smithi196 ceph-mon[76349]: from='client.67535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/118837105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:00.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:00.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:00.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='client.67541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='client.67547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1010117493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='client.67541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='client.67547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1010117493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='client.67541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='client.67547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:01.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1010117493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:02 smithi184 ceph-mon[100622]: from='client.57597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:02 smithi184 ceph-mon[102844]: from='client.57597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:12:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:02 smithi196 ceph-mon[76349]: from='client.57597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:03 smithi184 ceph-mon[100622]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:03 smithi184 ceph-mon[102844]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:03 smithi196 ceph-mon[76349]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:05 smithi184 ceph-mon[100622]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:05 smithi184 ceph-mon[102844]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:05 smithi196 ceph-mon[76349]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:07 smithi184 ceph-mon[100622]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:07 smithi184 ceph-mon[102844]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:07 smithi196 ceph-mon[76349]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:09.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:09 smithi184 ceph-mon[100622]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:09.349 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:09 smithi184 ceph-mon[102844]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:09 smithi196 ceph-mon[76349]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:10.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:10 smithi184 ceph-mon[100622]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:10.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:10 smithi184 ceph-mon[102844]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:10 smithi196 ceph-mon[76349]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:10.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:10.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:10.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:12:12.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:12 smithi184 ceph-mon[100622]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:12 smithi184 ceph-mon[102844]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:12 smithi196 ceph-mon[76349]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:14 smithi184 ceph-mon[102844]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:14 smithi184 ceph-mon[100622]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:14 smithi196 ceph-mon[76349]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:16 smithi184 ceph-mon[102844]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:16 smithi184 ceph-mon[100622]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:16 smithi196 ceph-mon[76349]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:18 smithi184 ceph-mon[100622]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:18 smithi184 ceph-mon[102844]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:18 smithi196 ceph-mon[76349]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:20 smithi196 ceph-mon[76349]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[100622]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:20.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:20.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:20.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[102844]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:12:23.094 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:22 smithi196 ceph-mon[76349]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:22 smithi184 ceph-mon[100622]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:22 smithi184 ceph-mon[102844]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:24.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:24 smithi196 ceph-mon[76349]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:24 smithi184 ceph-mon[100622]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:24 smithi184 ceph-mon[102844]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:26 smithi196 ceph-mon[76349]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:26 smithi184 ceph-mon[100622]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:26 smithi184 ceph-mon[102844]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:28 smithi196 ceph-mon[76349]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:28 smithi184 ceph-mon[100622]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:28 smithi184 ceph-mon[102844]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:29.965 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:29.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:30.903 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:12:31.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:30 smithi184 ceph-mon[100622]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:31.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:30 smithi184 ceph-mon[102844]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:31.030 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:30.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:31.031 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:30.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:31.031 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:30.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:30 smithi196 ceph-mon[76349]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:31.485 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:12:31.485 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 3h - - 2024-09-20T23:12:31.485 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T23:12:31.485 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 12s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:12:31.485 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 12s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 12s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 12s ago 3h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 12s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 12s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 12s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 12s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 12s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 12s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:12:31.486 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 12s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:12:31.487 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:12:31.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:12:31.818 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:12:31.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:12:31.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:12:31.818 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:12:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:12:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:12:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:12:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:12:32.107 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:12:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:31 smithi196 ceph-mon[76349]: from='client.57609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:31 smithi196 ceph-mon[76349]: from='client.67577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2224780454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[100622]: from='client.57609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[100622]: from='client.67577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2224780454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[102844]: from='client.57609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[102844]: from='client.67577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2224780454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:32.431 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:12:32.431 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:32.431 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:32.431 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:32.431 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[100622]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[100622]: from='client.57621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[100622]: from='client.67595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2598254938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[102844]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[102844]: from='client.57621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[102844]: from='client.67595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2598254938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:32 smithi196 ceph-mon[76349]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:32 smithi196 ceph-mon[76349]: from='client.57621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:32 smithi196 ceph-mon[76349]: from='client.67595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2598254938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:34 smithi184 ceph-mon[100622]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:34 smithi184 ceph-mon[102844]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:34 smithi196 ceph-mon[76349]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:36 smithi184 ceph-mon[100622]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:36 smithi184 ceph-mon[102844]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:37.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:36 smithi196 ceph-mon[76349]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:38 smithi184 ceph-mon[100622]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:38 smithi184 ceph-mon[102844]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:38 smithi196 ceph-mon[76349]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[100622]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[102844]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:40.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:40.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:40.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:41 smithi196 ceph-mon[76349]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:12:42.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:12:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:43 smithi184 ceph-mon[102844]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:43 smithi184 ceph-mon[100622]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:43 smithi196 ceph-mon[76349]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:44.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:45.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:45 smithi196 ceph-mon[76349]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:45 smithi184 ceph-mon[100622]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:45 smithi184 ceph-mon[102844]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:47 smithi184 ceph-mon[100622]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:47 smithi184 ceph-mon[102844]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:47.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:47 smithi196 ceph-mon[76349]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:49 smithi184 ceph-mon[100622]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:49 smithi184 ceph-mon[102844]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:49 smithi196 ceph-mon[76349]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:51.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:51 smithi184 ceph-mon[100622]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:51 smithi184 ceph-mon[102844]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:50.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:50.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:12:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:12:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:12:50.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:12:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:51 smithi196 ceph-mon[76349]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:52.145 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:12:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:12:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:53 smithi184 ceph-mon[100622]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:53 smithi184 ceph-mon[102844]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:53 smithi196 ceph-mon[76349]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:55 smithi184 ceph-mon[100622]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:55 smithi184 ceph-mon[102844]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:55 smithi196 ceph-mon[76349]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:57 smithi184 ceph-mon[100622]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:57 smithi184 ceph-mon[102844]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:57 smithi196 ceph-mon[76349]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:59 smithi184 ceph-mon[102844]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:59 smithi184 ceph-mon[100622]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:59 smithi196 ceph-mon[76349]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:12:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:00.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[102844]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[100622]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:00.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:00.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:00.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:01 smithi196 ceph-mon[76349]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:13:02.727 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:13:03.258 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 3h - - 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 3h - - 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 4s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 4s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 4s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 4s ago 3h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 2s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 4s ago 3h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 4s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 4s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:13:03.259 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 4s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 2s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 2s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 2s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 2s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 2s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:13:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 2s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:13:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:03 smithi196 ceph-mon[76349]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:13:03.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:03 smithi184 ceph-mon[100622]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:13:03.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:03 smithi184 ceph-mon[102844]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:13:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:13:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:13:03.893 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:13:03.893 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:13:03.893 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:13:03.894 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:13:04.222 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:13:04.222 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:04.222 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[100622]: from='client.67607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[100622]: from='client.67613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1828398503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[102844]: from='client.67607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[102844]: from='client.67613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1828398503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:04 smithi196 ceph-mon[76349]: from='client.67607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:04 smithi196 ceph-mon[76349]: from='client.67613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1828398503' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:05.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:05 smithi196 ceph-mon[76349]: from='client.67619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:05 smithi196 ceph-mon[76349]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:05.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:05 smithi196 ceph-mon[76349]: from='client.67631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2068550521' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[100622]: from='client.67619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[100622]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[100622]: from='client.67631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2068550521' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[102844]: from='client.67619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[102844]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:05.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[102844]: from='client.67631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:05.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2068550521' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:07 smithi184 ceph-mon[100622]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:07 smithi184 ceph-mon[102844]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:07 smithi196 ceph-mon[76349]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:09 smithi184 ceph-mon[100622]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:09 smithi184 ceph-mon[102844]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:09 smithi196 ceph-mon[76349]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:11.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:11 smithi184 ceph-mon[100622]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:11.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:11 smithi184 ceph-mon[102844]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:11.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:10.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:11.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:10.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:11.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:10.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:11 smithi196 ceph-mon[76349]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:12.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:13:13.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:13 smithi184 ceph-mon[100622]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:13.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:13 smithi184 ceph-mon[102844]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:13 smithi196 ceph-mon[76349]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:15.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:15 smithi184 ceph-mon[100622]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:15.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:15 smithi184 ceph-mon[102844]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:15.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:15 smithi196 ceph-mon[76349]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:17 smithi184 ceph-mon[100622]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:17 smithi184 ceph-mon[102844]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:17.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:17.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:17 smithi196 ceph-mon[76349]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:19 smithi184 ceph-mon[100622]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:19 smithi184 ceph-mon[102844]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:19 smithi196 ceph-mon[76349]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[100622]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-mon[102844]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:20 smithi196 ceph-mon[76349]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:20.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:20.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:20.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:22 smithi196 ceph-mon[76349]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:22 smithi184 ceph-mon[100622]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:22 smithi184 ceph-mon[102844]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:24 smithi184 ceph-mon[100622]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:24 smithi184 ceph-mon[102844]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:25.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:24 smithi196 ceph-mon[76349]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:26 smithi196 ceph-mon[76349]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:26 smithi184 ceph-mon[100622]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:26 smithi184 ceph-mon[102844]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:28 smithi196 ceph-mon[76349]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:28 smithi184 ceph-mon[100622]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:28 smithi184 ceph-mon[102844]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:30 smithi196 ceph-mon[76349]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:30 smithi184 ceph-mon[100622]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:30 smithi184 ceph-mon[102844]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:30.993Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:30.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:30.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:13:32.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:32 smithi184 ceph-mon[100622]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:32.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:32 smithi184 ceph-mon[102844]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:32 smithi196 ceph-mon[76349]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:34.514 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 3h - - 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 15s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 91.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 15s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 15s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 15s ago 3h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:13:35.063 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 15s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 15s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 15s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 15s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 15s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 15s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 15s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:13:35.064 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:13:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:34 smithi184 ceph-mon[100622]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:34 smithi184 ceph-mon[102844]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:35.375 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:13:35.376 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:13:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:34 smithi196 ceph-mon[76349]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:13:35.649 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:13:35.650 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:13:35.651 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:13:35.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[100622]: from='client.57681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:35.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[100622]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:35.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[100622]: from='client.67655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:35.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3076700635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:35.960 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:13:35.960 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:35.960 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[102844]: from='client.57681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[102844]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[102844]: from='client.67655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3076700635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:35 smithi196 ceph-mon[76349]: from='client.57681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:35 smithi196 ceph-mon[76349]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:35 smithi196 ceph-mon[76349]: from='client.67655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:36.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3076700635' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[100622]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[100622]: from='client.57705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1211243312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[102844]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[102844]: from='client.57705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1211243312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:36 smithi196 ceph-mon[76349]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:36 smithi196 ceph-mon[76349]: from='client.57705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1211243312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:38 smithi184 ceph-mon[100622]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:38 smithi184 ceph-mon[102844]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:38 smithi196 ceph-mon[76349]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:40 smithi196 ceph-mon[76349]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[102844]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-mon[100622]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:40.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:40.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:40.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:13:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:42 smithi184 ceph-mon[100622]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:42 smithi184 ceph-mon[102844]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:42 smithi196 ceph-mon[76349]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:44.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:45.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:45 smithi196 ceph-mon[76349]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:45 smithi184 ceph-mon[100622]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:45 smithi184 ceph-mon[102844]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:47.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:47 smithi196 ceph-mon[76349]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:47.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:47 smithi184 ceph-mon[100622]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:47 smithi184 ceph-mon[102844]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:49 smithi184 ceph-mon[100622]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:49 smithi184 ceph-mon[102844]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:49 smithi196 ceph-mon[76349]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:51 smithi184 ceph-mon[100622]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:51 smithi184 ceph-mon[102844]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:50.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:50.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:13:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:13:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:13:50.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:13:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:51 smithi196 ceph-mon[76349]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:13:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:13:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:53 smithi184 ceph-mon[102844]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:53 smithi184 ceph-mon[100622]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:53 smithi196 ceph-mon[76349]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:55 smithi184 ceph-mon[100622]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:55 smithi184 ceph-mon[102844]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:55 smithi196 ceph-mon[76349]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:57 smithi184 ceph-mon[102844]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:57 smithi184 ceph-mon[100622]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:57 smithi196 ceph-mon[76349]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:59 smithi184 ceph-mon[102844]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:59 smithi184 ceph-mon[100622]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:59 smithi196 ceph-mon[76349]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[100622]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[102844]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:00 smithi196 ceph-mon[76349]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:00.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:00.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:00.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:03 smithi196 ceph-mon[76349]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:03 smithi184 ceph-mon[100622]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:03 smithi184 ceph-mon[102844]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:05 smithi196 ceph-mon[76349]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:05.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:05 smithi184 ceph-mon[100622]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:05.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:05 smithi184 ceph-mon[102844]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:06.255 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 3h - - 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:14:06.783 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 3h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 3h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 5s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 5s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 5s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 5s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:14:06.784 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:14:06.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:14:06.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 5s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:14:06.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:14:06.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 5s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:07.092 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:14:07.093 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:14:07.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[100622]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:14:07.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:07.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4150735412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:07.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[102844]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:14:07.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:07.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4150735412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:14:07.362 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:14:07.363 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:14:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:07 smithi196 ceph-mon[76349]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:14:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4150735412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:07.674 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:14:07.674 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:07.674 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:08 smithi196 ceph-mon[76349]: from='client.57717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:08 smithi196 ceph-mon[76349]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:08 smithi196 ceph-mon[76349]: from='client.67691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1612232621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[100622]: from='client.57717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[100622]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[100622]: from='client.67691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1612232621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[102844]: from='client.57717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[102844]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[102844]: from='client.67691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1612232621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:09 smithi196 ceph-mon[76349]: from='client.67703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:09 smithi196 ceph-mon[76349]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:14:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:09 smithi184 ceph-mon[100622]: from='client.67703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:09 smithi184 ceph-mon[100622]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:14:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:09 smithi184 ceph-mon[102844]: from='client.67703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:09 smithi184 ceph-mon[102844]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:14:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:11 smithi184 ceph-mon[100622]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:14:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:11 smithi184 ceph-mon[102844]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:14:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:10.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:10.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:10 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:10.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:11 smithi196 ceph-mon[76349]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:14:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:14:13.342 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:13 smithi184 ceph-mon[100622]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:14:13.342 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:13 smithi184 ceph-mon[102844]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:14:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:13 smithi196 ceph-mon[76349]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:14:14.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:14.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:15 smithi184 ceph-mon[100622]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T23:14:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:15 smithi184 ceph-mon[102844]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T23:14:15.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:15 smithi196 ceph-mon[76349]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-20T23:14:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:17 smithi184 ceph-mon[100622]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:14:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:17 smithi184 ceph-mon[102844]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:14:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:17 smithi196 ceph-mon[76349]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:14:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:19.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:19 smithi184 ceph-mon[100622]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:14:19.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:19 smithi184 ceph-mon[102844]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:14:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:19 smithi196 ceph-mon[76349]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:14:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:20 smithi196 ceph-mon[76349]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:14:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[100622]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[102844]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:20.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:20.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:20 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:20.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[100622]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:22 smithi184 ceph-mon[102844]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:22 smithi196 ceph-mon[76349]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:14:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:24 smithi184 ceph-mon[100622]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:14:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:24 smithi184 ceph-mon[102844]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:14:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:24 smithi196 ceph-mon[76349]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:14:26.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:26 smithi196 ceph-mon[76349]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:14:26.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:26 smithi184 ceph-mon[100622]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:14:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:26 smithi184 ceph-mon[102844]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:14:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:28 smithi196 ceph-mon[76349]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:28 smithi184 ceph-mon[100622]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:14:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:28 smithi184 ceph-mon[102844]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:14:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:30.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:30 smithi184 ceph-mon[100622]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:14:30.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:30 smithi184 ceph-mon[102844]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:14:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:30 smithi196 ceph-mon[76349]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:14:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:30.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:30.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:30 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:30.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:14:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:32 smithi184 ceph-mon[100622]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:14:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:32 smithi184 ceph-mon[102844]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:14:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:32 smithi196 ceph-mon[76349]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:14:35.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:34 smithi184 ceph-mon[100622]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:14:35.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:34 smithi184 ceph-mon[102844]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:14:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:34 smithi196 ceph-mon[76349]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:14:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:36 smithi184 ceph-mon[100622]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T23:14:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:36 smithi184 ceph-mon[102844]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T23:14:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:36 smithi196 ceph-mon[76349]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-20T23:14:37.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:37.969 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 18s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 18s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 18s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 3h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:14:38.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 18s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 18s ago 3h 78.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 18s ago 3h 81.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 18s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 3h 88.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 3h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 3h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 3h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:14:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:14:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:38 smithi184 ceph-mon[100622]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T23:14:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:38 smithi184 ceph-mon[100622]: from='client.57753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:38 smithi184 ceph-mon[102844]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T23:14:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:38 smithi184 ceph-mon[102844]: from='client.57753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:14:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:14:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:14:38.808 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:14:39.077 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:14:39.078 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:14:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:38 smithi196 ceph-mon[76349]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T23:14:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:38 smithi196 ceph-mon[76349]: from='client.57753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:39.389 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:14:39.389 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:39.389 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:39.389 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:39.389 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[100622]: from='client.67721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[100622]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1563314951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[100622]: from='client.67739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/587932396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[102844]: from='client.67721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[102844]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1563314951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[102844]: from='client.67739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/587932396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:39 smithi196 ceph-mon[76349]: from='client.67721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:39 smithi196 ceph-mon[76349]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1563314951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:39 smithi196 ceph-mon[76349]: from='client.67739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/587932396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[100622]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:14:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[102844]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:40.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:40.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:41.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:40 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:40.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:40 smithi196 ceph-mon[76349]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:14:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:14:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:42 smithi196 ceph-mon[76349]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:14:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[100622]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:14:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:42 smithi184 ceph-mon[102844]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:14:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:44 smithi196 ceph-mon[76349]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T23:14:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:44 smithi184 ceph-mon[100622]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T23:14:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:44 smithi184 ceph-mon[102844]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T23:14:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:46 smithi196 ceph-mon[76349]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:14:46.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:46 smithi184 ceph-mon[100622]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:14:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:46 smithi184 ceph-mon[102844]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-20T23:14:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:48 smithi196 ceph-mon[76349]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:48 smithi184 ceph-mon[100622]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:48 smithi184 ceph-mon[102844]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:50 smithi196 ceph-mon[76349]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:51.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:50 smithi184 ceph-mon[100622]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:51.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:50 smithi184 ceph-mon[102844]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:50.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:50 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:50.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:14:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:14:51.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:14:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:14:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:14:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:52 smithi196 ceph-mon[76349]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:52 smithi184 ceph-mon[100622]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:52 smithi184 ceph-mon[102844]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:54 smithi184 ceph-mon[100622]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:54 smithi184 ceph-mon[102844]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:54 smithi196 ceph-mon[76349]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:56 smithi184 ceph-mon[100622]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:56 smithi184 ceph-mon[102844]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:56 smithi196 ceph-mon[76349]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:58 smithi196 ceph-mon[76349]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:58 smithi184 ceph-mon[100622]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:58 smithi184 ceph-mon[102844]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:59.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:59.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:59.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:00 smithi196 ceph-mon[76349]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[102844]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[100622]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:00.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:01.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:01.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:01.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:15:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[100622]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[102844]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:02 smithi196 ceph-mon[76349]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:04 smithi184 ceph-mon[100622]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:04.802 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:04 smithi184 ceph-mon[102844]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:04 smithi196 ceph-mon[76349]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:06 smithi196 ceph-mon[76349]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:06 smithi184 ceph-mon[102844]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:06 smithi184 ceph-mon[100622]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:08.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:08 smithi196 ceph-mon[76349]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:08 smithi184 ceph-mon[100622]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:08 smithi184 ceph-mon[102844]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:09.690 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 8s ago 3h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:15:10.215 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 8s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 3h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 8s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 3h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 8s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 8s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:15:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 8s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:15:10.217 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 8s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:15:10.217 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:15:10.217 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 8s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:15:10.217 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:15:10.217 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 8s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:15:10.525 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:10.526 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:15:10.527 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:15:10.527 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:15:10.527 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:15:10.794 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:15:10.794 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:15:10.794 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:15:10.794 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:15:10.794 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:15:10.795 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:15:10.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[100622]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:10.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[100622]: from='client.57789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[100622]: from='client.67757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3277266013' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[102844]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[102844]: from='client.57789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[102844]: from='client.67757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3277266013' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:10 smithi196 ceph-mon[76349]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:10 smithi196 ceph-mon[76349]: from='client.57789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:10 smithi196 ceph-mon[76349]: from='client.67757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3277266013' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:11.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:11.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:11.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:11.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:11.103 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:15:11.103 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:11.103 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:11 smithi196 ceph-mon[76349]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:11 smithi196 ceph-mon[76349]: from='client.57813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3453199489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[102844]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[102844]: from='client.57813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3453199489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[100622]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[100622]: from='client.57813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3453199489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:15:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:12 smithi196 ceph-mon[76349]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:12 smithi184 ceph-mon[100622]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:12 smithi184 ceph-mon[102844]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:14 smithi184 ceph-mon[100622]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:14 smithi184 ceph-mon[102844]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:14 smithi196 ceph-mon[76349]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:16 smithi184 ceph-mon[100622]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:16 smithi184 ceph-mon[102844]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:16 smithi196 ceph-mon[76349]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:18.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:18 smithi196 ceph-mon[76349]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:18 smithi184 ceph-mon[100622]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:18 smithi184 ceph-mon[102844]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[100622]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[102844]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:21.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:21.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:21.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:21.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:21.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:21 smithi196 ceph-mon[76349]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:22 smithi196 ceph-mon[76349]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[100622]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[102844]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:24 smithi196 ceph-mon[76349]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:24 smithi184 ceph-mon[100622]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:24 smithi184 ceph-mon[102844]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:26 smithi196 ceph-mon[76349]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:26 smithi184 ceph-mon[100622]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:27.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:26 smithi184 ceph-mon[102844]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:28 smithi196 ceph-mon[76349]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:28 smithi184 ceph-mon[100622]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:28 smithi184 ceph-mon[102844]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:30 smithi196 ceph-mon[76349]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:31.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:30 smithi184 ceph-mon[100622]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:31.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:30 smithi184 ceph-mon[102844]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:31.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:31.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:31.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:15:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:32 smithi184 ceph-mon[102844]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:33.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:32 smithi184 ceph-mon[100622]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:32 smithi196 ceph-mon[76349]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:34 smithi184 ceph-mon[100622]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:34 smithi184 ceph-mon[102844]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:34 smithi196 ceph-mon[76349]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:37.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:36 smithi184 ceph-mon[100622]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:37.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:37.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:36 smithi184 ceph-mon[102844]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:37.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:36 smithi196 ceph-mon[76349]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:38 smithi184 ceph-mon[100622]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:38 smithi184 ceph-mon[102844]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:38 smithi196 ceph-mon[76349]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[102844]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[100622]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:41.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:41.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:41.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:41.404 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:15:41.610 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:41 smithi196 ceph-mon[76349]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:41.610 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:41.931 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:15:41.931 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 3h - - 2024-09-20T23:15:41.931 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T23:15:41.931 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 1s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:15:41.931 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 3h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 1s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 1s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 1s ago 3h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 0s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 1s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 1s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 0s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 1s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 1s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 1s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:15:41.932 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 0s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 0s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 0s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 0s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:15:41.933 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:15:42.239 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:15:42.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:15:42.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:15:42.240 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:15:42.241 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:15:42.509 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:15:42.510 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:15:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='client.57825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='client.57831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1954758294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='client.57825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='client.57831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:42.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:42.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1954758294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:42.823 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:15:42.824 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:42.824 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:42.824 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:42.824 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='client.57825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='client.57831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:15:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:43.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1954758294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:43 smithi184 ceph-mon[100622]: from='client.67811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:44.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4180146534' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:43 smithi184 ceph-mon[102844]: from='client.67811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4180146534' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:43 smithi196 ceph-mon[76349]: from='client.67811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4180146534' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:44 smithi184 ceph-mon[102844]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:44 smithi184 ceph-mon[100622]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:44 smithi196 ceph-mon[76349]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:46 smithi184 ceph-mon[100622]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:46 smithi184 ceph-mon[102844]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:46 smithi196 ceph-mon[76349]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:48 smithi196 ceph-mon[76349]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:48 smithi184 ceph-mon[100622]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:48 smithi184 ceph-mon[102844]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:50.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:50 smithi196 ceph-mon[76349]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:50 smithi184 ceph-mon[100622]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:50 smithi184 ceph-mon[102844]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:51.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:51.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:15:51.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:15:51.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:15:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:15:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:15:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:52 smithi184 ceph-mon[100622]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:52 smithi184 ceph-mon[102844]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:52 smithi196 ceph-mon[76349]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:54 smithi184 ceph-mon[100622]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:54 smithi184 ceph-mon[102844]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:54 smithi196 ceph-mon[76349]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:56.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:56 smithi184 ceph-mon[100622]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:56.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:56.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:56 smithi184 ceph-mon[102844]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:56.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:56 smithi196 ceph-mon[76349]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:58.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:58 smithi184 ceph-mon[100622]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:58.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:58 smithi184 ceph-mon[102844]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:58 smithi196 ceph-mon[76349]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[102844]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[100622]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:01.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:01.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:01.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:01.050 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:01.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:00 smithi196 ceph-mon[76349]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:16:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:02 smithi196 ceph-mon[76349]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[100622]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[102844]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:04 smithi184 ceph-mon[100622]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:04 smithi184 ceph-mon[102844]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:04 smithi196 ceph-mon[76349]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:06 smithi184 ceph-mon[100622]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:06 smithi184 ceph-mon[102844]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:06 smithi196 ceph-mon[76349]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:09.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:08 smithi196 ceph-mon[76349]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:08 smithi184 ceph-mon[100622]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:08 smithi184 ceph-mon[102844]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:10 smithi196 ceph-mon[76349]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:10 smithi184 ceph-mon[100622]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:10 smithi184 ceph-mon[102844]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:11.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:11.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:11.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:16:13.118 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:16:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:12 smithi184 ceph-mon[100622]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:12 smithi184 ceph-mon[102844]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:13.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:12 smithi196 ceph-mon[76349]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:13.645 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 3h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:16:13.646 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 13s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 13s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:16:13.647 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:16:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:16:13.961 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:16:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:16:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:16:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:16:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:16:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:16:14.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:16:14.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:13 smithi184 ceph-mon[100622]: from='client.57861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:14.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:13 smithi184 ceph-mon[102844]: from='client.57861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:13 smithi196 ceph-mon[76349]: from='client.57861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:14.538 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:16:14.539 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:14.539 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: from='client.57867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: from='client.57873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/819329527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1544656248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: from='client.57867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: from='client.57873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/819329527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1544656248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: from='client.57867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: from='client.57873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/819329527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1544656248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:16.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:15 smithi196 ceph-mon[76349]: from='client.67847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:15 smithi184 ceph-mon[100622]: from='client.67847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:15 smithi184 ceph-mon[102844]: from='client.67847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:16 smithi184 ceph-mon[100622]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:16 smithi184 ceph-mon[102844]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:16 smithi196 ceph-mon[76349]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:18 smithi184 ceph-mon[100622]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:18 smithi184 ceph-mon[102844]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:18 smithi196 ceph-mon[76349]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[100622]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[102844]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:21.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:21.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:21.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:20 smithi196 ceph-mon[76349]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:22] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[100622]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[102844]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:23 smithi196 ceph-mon[76349]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:25 smithi184 ceph-mon[100622]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:25 smithi184 ceph-mon[102844]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:25 smithi196 ceph-mon[76349]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:27 smithi184 ceph-mon[102844]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:27 smithi184 ceph-mon[100622]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:27 smithi196 ceph-mon[76349]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:29 smithi184 ceph-mon[100622]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:29 smithi184 ceph-mon[102844]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:29 smithi196 ceph-mon[76349]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:31.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:31 smithi196 ceph-mon[76349]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:31 smithi184 ceph-mon[100622]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:31 smithi184 ceph-mon[102844]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:31.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:31.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:31.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:16:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:33 smithi184 ceph-mon[100622]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:33 smithi184 ceph-mon[102844]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:33 smithi196 ceph-mon[76349]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:35 smithi184 ceph-mon[100622]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:35 smithi184 ceph-mon[102844]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:35 smithi196 ceph-mon[76349]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:37 smithi184 ceph-mon[100622]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:37 smithi184 ceph-mon[102844]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:37 smithi196 ceph-mon[76349]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:39 smithi184 ceph-mon[100622]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:39 smithi184 ceph-mon[102844]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:39 smithi196 ceph-mon[76349]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[100622]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[102844]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:41.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:41.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:41.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:41.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:41 smithi196 ceph-mon[76349]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:16:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[100622]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[102844]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:43 smithi196 ceph-mon[76349]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:16:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:44.833 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 3h - - 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 4s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 4s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 4s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 4s ago 3h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:16:45.363 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 3s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 4s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 4s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 4s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 4s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 3s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 3s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 3s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 3s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:16:45.364 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:16:45.365 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 3s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:16:45.365 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:16:45.365 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 3s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:16:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:45 smithi184 ceph-mon[100622]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:45 smithi184 ceph-mon[102844]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:45 smithi196 ceph-mon[76349]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:16:45.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:16:45.677 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:16:45.946 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:16:45.947 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:16:45.947 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:16:45.947 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:16:45.947 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:16:45.947 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:16:46.259 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:16:46.259 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:46.259 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[102844]: from='client.67859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[102844]: from='client.67865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3980365553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[100622]: from='client.67859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[100622]: from='client.67865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3980365553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:46.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:46 smithi196 ceph-mon[76349]: from='client.67859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:46 smithi196 ceph-mon[76349]: from='client.67865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:46.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3980365553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[102844]: from='client.57909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[102844]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[102844]: from='client.57921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3186508273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[100622]: from='client.57909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[100622]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[100622]: from='client.57921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3186508273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:47.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:47 smithi196 ceph-mon[76349]: from='client.57909 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:47 smithi196 ceph-mon[76349]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:47.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:47.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:47 smithi196 ceph-mon[76349]: from='client.57921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:47.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3186508273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:49 smithi184 ceph-mon[100622]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:49 smithi184 ceph-mon[102844]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:49 smithi196 ceph-mon[76349]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:51.163 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:51.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:51.163 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:51.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:16:51.163 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:16:51.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:16:51.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:51 smithi184 ceph-mon[100622]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:51.461 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:51 smithi184 ceph-mon[102844]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:51.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:51 smithi196 ceph-mon[76349]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:16:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:16:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:53 smithi196 ceph-mon[76349]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:53 smithi184 ceph-mon[100622]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:53 smithi184 ceph-mon[102844]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:55 smithi184 ceph-mon[100622]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:55 smithi184 ceph-mon[102844]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:55 smithi196 ceph-mon[76349]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:57 smithi184 ceph-mon[100622]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:57 smithi184 ceph-mon[102844]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:57 smithi196 ceph-mon[76349]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:59 smithi184 ceph-mon[100622]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:59 smithi184 ceph-mon[102844]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:59 smithi196 ceph-mon[76349]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:01.178 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:01.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:01.178 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:01.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:01.178 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:01.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[100622]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[102844]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:01 smithi196 ceph-mon[76349]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:17:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[100622]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[102844]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:03 smithi196 ceph-mon[76349]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:05 smithi184 ceph-mon[100622]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:05 smithi184 ceph-mon[102844]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:05 smithi196 ceph-mon[76349]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:07 smithi184 ceph-mon[100622]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:07 smithi184 ceph-mon[102844]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:07 smithi196 ceph-mon[76349]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:09 smithi184 ceph-mon[100622]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:09 smithi184 ceph-mon[102844]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:09 smithi196 ceph-mon[76349]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:11 smithi184 ceph-mon[100622]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:11 smithi184 ceph-mon[102844]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:11.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:11.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:11.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:11.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:11 smithi196 ceph-mon[76349]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:17:13.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:13 smithi196 ceph-mon[76349]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:13 smithi184 ceph-mon[100622]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:13 smithi184 ceph-mon[102844]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:15 smithi184 ceph-mon[100622]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:15 smithi184 ceph-mon[102844]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:15 smithi196 ceph-mon[76349]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:16.558 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:17:17.081 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:17:17.081 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 3h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:17:17.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:17:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:17:17.084 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:17:17.084 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:17:17.084 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:17:17.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:17.397 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:17:17.397 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:17:17.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:17:17.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:17:17.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:17 smithi184 ceph-mon[100622]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:17.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:17.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:17 smithi184 ceph-mon[102844]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:17.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:17 smithi196 ceph-mon[76349]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:17.665 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:17:17.666 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:17:17.667 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:17:17.978 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:17:17.978 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:17.978 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:17.978 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:17.978 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[102844]: from='client.67895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[102844]: from='client.57939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[102844]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/965826479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1721697003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[100622]: from='client.67895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[100622]: from='client.57939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[100622]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/965826479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:18.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1721697003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:18.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:18 smithi196 ceph-mon[76349]: from='client.67895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:18 smithi196 ceph-mon[76349]: from='client.57939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:18 smithi196 ceph-mon[76349]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:18.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/965826479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:18.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1721697003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:19 smithi184 ceph-mon[100622]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:19 smithi184 ceph-mon[100622]: from='client.67919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:19 smithi184 ceph-mon[102844]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:19 smithi184 ceph-mon[102844]: from='client.67919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:19 smithi196 ceph-mon[76349]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:19 smithi196 ceph-mon[76349]: from='client.67919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[100622]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[102844]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:21.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:21.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:21.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:21.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:21 smithi196 ceph-mon[76349]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:17:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[102844]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[100622]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:23 smithi196 ceph-mon[76349]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:25 smithi184 ceph-mon[100622]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:25 smithi184 ceph-mon[102844]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:25 smithi196 ceph-mon[76349]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:26 smithi184 ceph-mon[100622]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:26 smithi184 ceph-mon[102844]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:26 smithi196 ceph-mon[76349]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:28 smithi196 ceph-mon[76349]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:28 smithi184 ceph-mon[100622]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:28 smithi184 ceph-mon[102844]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:30 smithi196 ceph-mon[76349]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:31.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:30 smithi184 ceph-mon[100622]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:31.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:30 smithi184 ceph-mon[102844]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:31.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:31.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:31.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:17:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:32 smithi196 ceph-mon[76349]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:32.971 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:32 smithi184 ceph-mon[100622]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:32.972 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:32 smithi184 ceph-mon[102844]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:34 smithi196 ceph-mon[76349]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:34 smithi184 ceph-mon[100622]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:34 smithi184 ceph-mon[102844]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:36 smithi196 ceph-mon[76349]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:36 smithi184 ceph-mon[102844]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:36 smithi184 ceph-mon[100622]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:38 smithi196 ceph-mon[76349]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:38 smithi184 ceph-mon[100622]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:38 smithi184 ceph-mon[102844]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:40 smithi196 ceph-mon[76349]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:40.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:40 smithi184 ceph-mon[100622]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:40.910 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:40 smithi184 ceph-mon[102844]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:41.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:41.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:41.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:17:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[100622]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[102844]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:42 smithi196 ceph-mon[76349]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:43.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:17:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:44 smithi184 ceph-mon[100622]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:44 smithi184 ceph-mon[102844]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:44 smithi196 ceph-mon[76349]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:46 smithi184 ceph-mon[100622]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:46 smithi184 ceph-mon[102844]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:46 smithi196 ceph-mon[76349]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:17:48.804 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 6s ago 3h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 3h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 6s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 6s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:17:48.805 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 6s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 6s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 6s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:17:48.806 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 6s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:17:48.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:17:48.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 6s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:17:49.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:48 smithi184 ceph-mon[100622]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:49.119 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:48 smithi184 ceph-mon[102844]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:49.119 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:17:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:17:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:17:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:49.119 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:17:49.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:17:49.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:48 smithi196 ceph-mon[76349]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:17:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:17:49.387 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:17:49.696 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:17:49.697 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:49.697 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:49.697 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:49.697 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[100622]: from='client.67931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[100622]: from='client.67937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[100622]: from='client.67943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1937850255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/828574701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[102844]: from='client.67931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[102844]: from='client.67937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[102844]: from='client.67943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1937850255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/828574701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:49 smithi196 ceph-mon[76349]: from='client.67931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:49 smithi196 ceph-mon[76349]: from='client.67937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:49 smithi196 ceph-mon[76349]: from='client.67943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1937850255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/828574701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-mon[100622]: from='client.67955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-mon[100622]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-mon[102844]: from='client.67955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-mon[102844]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:51.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:51.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:17:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:17:51.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:17:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:51 smithi196 ceph-mon[76349]: from='client.67955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:51 smithi196 ceph-mon[76349]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:17:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:17:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:53 smithi184 ceph-mon[100622]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:53 smithi184 ceph-mon[102844]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:53 smithi196 ceph-mon[76349]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:55 smithi184 ceph-mon[100622]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:55 smithi184 ceph-mon[102844]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:55 smithi196 ceph-mon[76349]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:57 smithi184 ceph-mon[100622]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:57 smithi184 ceph-mon[102844]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:57 smithi196 ceph-mon[76349]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:59 smithi196 ceph-mon[76349]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:59 smithi184 ceph-mon[100622]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:59 smithi184 ceph-mon[102844]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:01.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:01 smithi184 ceph-mon[100622]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:01.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:01 smithi184 ceph-mon[102844]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:01.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:01.013Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:01.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:01.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:01.284 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:01.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:01 smithi196 ceph-mon[76349]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:02.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.264 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:18:02.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:03 smithi196 ceph-mon[76349]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[100622]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[102844]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:03.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:05 smithi184 ceph-mon[100622]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:05 smithi184 ceph-mon[102844]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:05 smithi196 ceph-mon[76349]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:07 smithi196 ceph-mon[76349]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:07 smithi184 ceph-mon[100622]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:07 smithi184 ceph-mon[102844]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:09 smithi196 ceph-mon[76349]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:09 smithi184 ceph-mon[100622]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:09 smithi184 ceph-mon[102844]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:11 smithi184 ceph-mon[100622]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:11 smithi184 ceph-mon[102844]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:11.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:11.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:11.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:11 smithi196 ceph-mon[76349]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:18:13.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:13 smithi196 ceph-mon[76349]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:13 smithi184 ceph-mon[100622]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:13 smithi184 ceph-mon[102844]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:14.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:18:13.81266861Z level=info msg="Completed cleanup jobs" duration=13.080748ms 2024-09-20T23:18:14.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:15.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:15 smithi196 ceph-mon[76349]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:15 smithi184 ceph-mon[100622]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:15 smithi184 ceph-mon[102844]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:17 smithi196 ceph-mon[76349]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:17 smithi184 ceph-mon[100622]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:17 smithi184 ceph-mon[102844]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:19 smithi184 ceph-mon[100622]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:19 smithi184 ceph-mon[102844]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:19 smithi196 ceph-mon[76349]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:19.999 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:18:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:18:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 3h - - 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 19s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 3h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 19s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:18:20.527 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:18:20.528 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:18:20.838 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:18:20.838 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:18:20.838 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:18:20.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:20.838 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:18:20.839 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:18:21.109 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:18:21.110 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:18:21.110 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:21.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:21.110 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:21.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:21.110 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:21.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:21.458 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:18:21.458 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:21.458 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:21.458 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:21.458 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:21.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[100622]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:21.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[100622]: from='client.67967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:21.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3527676759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:21.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[102844]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:21.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[102844]: from='client.67967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:21.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3527676759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:21 smithi196 ceph-mon[76349]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:21 smithi196 ceph-mon[76349]: from='client.67967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3527676759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='client.67973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='client.67991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2187915949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='client.67973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='client.67991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2187915949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='client.67973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='client.67991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:22.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2187915949' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[102844]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[100622]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:23 smithi196 ceph-mon[76349]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:25 smithi184 ceph-mon[100622]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:25 smithi184 ceph-mon[102844]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:25 smithi196 ceph-mon[76349]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:26 smithi184 ceph-mon[100622]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:26 smithi184 ceph-mon[102844]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:26 smithi196 ceph-mon[76349]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:28 smithi184 ceph-mon[100622]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:28 smithi184 ceph-mon[102844]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:28 smithi196 ceph-mon[76349]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:30 smithi184 ceph-mon[100622]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:30 smithi184 ceph-mon[102844]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:31.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:31.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:31.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:31.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:30 smithi196 ceph-mon[76349]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:18:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:32 smithi184 ceph-mon[100622]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:32 smithi184 ceph-mon[102844]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:32 smithi196 ceph-mon[76349]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:34 smithi184 ceph-mon[100622]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:34 smithi184 ceph-mon[102844]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:35.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:34 smithi196 ceph-mon[76349]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:36 smithi184 ceph-mon[100622]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:36 smithi184 ceph-mon[102844]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:36 smithi196 ceph-mon[76349]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:38 smithi184 ceph-mon[100622]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:38 smithi184 ceph-mon[102844]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:38 smithi196 ceph-mon[76349]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:40 smithi184 ceph-mon[100622]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:40 smithi184 ceph-mon[102844]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:41.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:41.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:41.049 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:41.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:40 smithi196 ceph-mon[76349]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:42.259 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:18:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[100622]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:42 smithi184 ceph-mon[102844]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:42 smithi196 ceph-mon[76349]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:44.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.237 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.237 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:18:45.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:44 smithi184 ceph-mon[100622]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:45.204 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:45.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:44 smithi184 ceph-mon[102844]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:45.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:44 smithi196 ceph-mon[76349]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:47.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:46 smithi196 ceph-mon[76349]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:47.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:46 smithi184 ceph-mon[102844]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:46 smithi184 ceph-mon[100622]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:48 smithi184 ceph-mon[100622]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:48 smithi184 ceph-mon[102844]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:48 smithi196 ceph-mon[76349]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:50 smithi184 ceph-mon[100622]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:50 smithi184 ceph-mon[102844]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:51.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:51.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:18:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:18:51.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:18:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:50 smithi196 ceph-mon[76349]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:51.757 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:18:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:18:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:18:52.285 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 3h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:18:52.286 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:18:52.287 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:18:52.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:18:52.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:18:52.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:18:52.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:18:52.600 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:18:52.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:52.601 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:18:52.602 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:18:52.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:18:52.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:18:52.603 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:18:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:18:52.869 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:18:53.178 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:18:53.178 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:53.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:53.179 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:53.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:53.179 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[100622]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[100622]: from='client.58041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[100622]: from='client.68009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2282067570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[102844]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[102844]: from='client.58041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[102844]: from='client.68009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2282067570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:52 smithi196 ceph-mon[76349]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:52 smithi196 ceph-mon[76349]: from='client.58041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:52 smithi196 ceph-mon[76349]: from='client.68009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2282067570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[100622]: from='client.68015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[100622]: from='client.68027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1884377130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[102844]: from='client.68015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[102844]: from='client.68027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1884377130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:53 smithi196 ceph-mon[76349]: from='client.68015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:53 smithi196 ceph-mon[76349]: from='client.68027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1884377130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:54 smithi196 ceph-mon[76349]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:54 smithi184 ceph-mon[100622]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:54 smithi184 ceph-mon[102844]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:57.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:56 smithi196 ceph-mon[76349]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:57.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:56 smithi184 ceph-mon[100622]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:56 smithi184 ceph-mon[102844]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:58 smithi184 ceph-mon[100622]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:18:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:58 smithi184 ceph-mon[102844]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:18:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:58 smithi196 ceph-mon[76349]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:18:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:00 smithi184 ceph-mon[100622]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:00 smithi184 ceph-mon[102844]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:01.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:01.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:01.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:00 smithi196 ceph-mon[76349]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[102844]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:02 smithi184 ceph-mon[100622]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:02 smithi196 ceph-mon[76349]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:05.359 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:05 smithi184 ceph-mon[100622]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:05.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:05 smithi184 ceph-mon[102844]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:05 smithi196 ceph-mon[76349]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:07 smithi196 ceph-mon[76349]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:07 smithi184 ceph-mon[100622]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:07 smithi184 ceph-mon[102844]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:09 smithi196 ceph-mon[76349]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:09 smithi184 ceph-mon[100622]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:09 smithi184 ceph-mon[102844]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:11 smithi184 ceph-mon[100622]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:11 smithi184 ceph-mon[102844]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:11.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:11.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:11.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:11 smithi196 ceph-mon[76349]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:19:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:13 smithi184 ceph-mon[100622]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:13 smithi184 ceph-mon[102844]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:13 smithi196 ceph-mon[76349]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:15.364 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:15 smithi196 ceph-mon[76349]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:15 smithi184 ceph-mon[100622]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:15 smithi184 ceph-mon[102844]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:17 smithi196 ceph-mon[76349]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:17 smithi184 ceph-mon[100622]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:17 smithi184 ceph-mon[102844]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:19 smithi196 ceph-mon[76349]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:19 smithi184 ceph-mon[100622]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:19 smithi184 ceph-mon[102844]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:21 smithi184 ceph-mon[100622]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:21 smithi184 ceph-mon[102844]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:21.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:21.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:21.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:21 smithi196 ceph-mon[76349]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:19:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:22 smithi196 ceph-mon[76349]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[100622]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[102844]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:23.474 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:19:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:19:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T23:19:24.017 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 2s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 3h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 3h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 0s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 2s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 0s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:19:24.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 0s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 0s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 0s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 0s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 2s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:19:24.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:19:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:19:24.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:19:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.610 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:19:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:19:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:19:24.610 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:19:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:19:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:24.968 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:19:24.968 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:24.968 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:24.968 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:24.968 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: from='client.68039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: from='client.68045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: from='client.68051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2733245540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3312684642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: from='client.68039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: from='client.68045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: from='client.68051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2733245540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3312684642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: from='client.68039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: from='client.68045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: from='client.68051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:25.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2733245540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:25.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3312684642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:26 smithi184 ceph-mon[100622]: from='client.58101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:26 smithi184 ceph-mon[102844]: from='client.58101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:26 smithi196 ceph-mon[76349]: from='client.58101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:27 smithi184 ceph-mon[100622]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:27 smithi184 ceph-mon[102844]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:27 smithi196 ceph-mon[76349]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:29 smithi196 ceph-mon[76349]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:29 smithi184 ceph-mon[100622]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:29 smithi184 ceph-mon[102844]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:31 smithi184 ceph-mon[100622]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:31 smithi184 ceph-mon[102844]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:31.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:31.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:31.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:31 smithi196 ceph-mon[76349]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:19:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:33 smithi184 ceph-mon[100622]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:33 smithi184 ceph-mon[102844]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:33 smithi196 ceph-mon[76349]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:35 smithi184 ceph-mon[100622]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:35 smithi184 ceph-mon[102844]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:35 smithi196 ceph-mon[76349]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:37 smithi196 ceph-mon[76349]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:37.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:37 smithi184 ceph-mon[100622]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:37 smithi184 ceph-mon[102844]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:39 smithi184 ceph-mon[100622]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:39 smithi184 ceph-mon[102844]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:39 smithi196 ceph-mon[76349]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:41 smithi184 ceph-mon[100622]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:41 smithi184 ceph-mon[102844]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:41.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:41.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:41.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:41 smithi196 ceph-mon[76349]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[100622]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[102844]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:19:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:42 smithi196 ceph-mon[76349]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[100622]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:44.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:44.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[102844]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:44.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:44 smithi196 ceph-mon[76349]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:46.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:46 smithi184 ceph-mon[100622]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:46.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:46.779 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:46 smithi184 ceph-mon[102844]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:46.779 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:46 smithi196 ceph-mon[76349]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:48 smithi184 ceph-mon[102844]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:48 smithi184 ceph-mon[100622]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:48 smithi196 ceph-mon[76349]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:51.023 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:50 smithi184 ceph-mon[100622]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:51.023 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:50 smithi184 ceph-mon[102844]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:50 smithi196 ceph-mon[76349]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:51.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:51.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:19:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:19:51.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:19:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:19:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:19:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:52 smithi184 ceph-mon[100622]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:52 smithi184 ceph-mon[102844]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:52 smithi196 ceph-mon[76349]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:54 smithi184 ceph-mon[100622]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:54 smithi184 ceph-mon[102844]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:54 smithi196 ceph-mon[76349]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:55.265 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:19:55.797 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:19:55.797 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T23:19:55.797 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 3h - - 2024-09-20T23:19:55.797 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 3h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:19:55.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:19:55.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:19:56.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:19:56.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:19:56.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:19:56.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:19:56.107 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:19:56.376 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:19:56.377 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:19:56.687 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:19:56.687 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:56.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:56.688 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:56.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:56.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: from='client.58113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:56.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: from='client.58119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:56.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:56.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: from='client.58125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:56.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:56.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4144930681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: from='client.58113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: from='client.58119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: from='client.58125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4144930681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: from='client.58113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: from='client.58119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: from='client.58125 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4144930681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:57.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:57 smithi196 ceph-mon[76349]: from='client.68099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:57.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2919771991' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:57 smithi184 ceph-mon[100622]: from='client.68099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:58.077 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2919771991' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:58.078 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:57 smithi184 ceph-mon[102844]: from='client.68099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:58.078 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2919771991' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:58 smithi184 ceph-mon[100622]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:58 smithi184 ceph-mon[102844]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:58 smithi196 ceph-mon[76349]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:00.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:00 smithi184 ceph-mon[100622]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:01.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:00 smithi184 ceph-mon[102844]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:01.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:00 smithi196 ceph-mon[76349]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:01.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:01.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:01.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:02.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:20:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[102844]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[100622]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:02 smithi196 ceph-mon[76349]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[100622]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:04 smithi184 ceph-mon[102844]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:04 smithi196 ceph-mon[76349]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:06 smithi184 ceph-mon[100622]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:06 smithi184 ceph-mon[102844]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:06 smithi196 ceph-mon[76349]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:08 smithi184 ceph-mon[100622]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:08 smithi184 ceph-mon[102844]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:08 smithi196 ceph-mon[76349]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:11.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:10 smithi184 ceph-mon[100622]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:11.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:10 smithi184 ceph-mon[102844]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:10 smithi196 ceph-mon[76349]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:11.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:11.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:11.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:20:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:12 smithi184 ceph-mon[100622]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:12 smithi184 ceph-mon[102844]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:12 smithi196 ceph-mon[76349]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:14 smithi184 ceph-mon[100622]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:14 smithi184 ceph-mon[102844]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:14 smithi196 ceph-mon[76349]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:17.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:16 smithi196 ceph-mon[76349]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:17.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:16 smithi184 ceph-mon[100622]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:16 smithi184 ceph-mon[102844]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:18 smithi184 ceph-mon[100622]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:18 smithi184 ceph-mon[102844]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:19.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:18 smithi196 ceph-mon[76349]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:21.026 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:20 smithi184 ceph-mon[100622]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:21.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:20 smithi184 ceph-mon[102844]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:21.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:20 smithi196 ceph-mon[76349]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:21.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:21.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:21.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:20:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[100622]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[102844]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:23 smithi196 ceph-mon[76349]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:24 smithi196 ceph-mon[76349]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[100622]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:24 smithi184 ceph-mon[102844]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:26.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:26 smithi184 ceph-mon[100622]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:26.937 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:26.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:26 smithi184 ceph-mon[102844]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:26.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:27.010 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:20:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:26 smithi196 ceph-mon[76349]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:20:27.555 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 3h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 3h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:20:27.556 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:20:27.557 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:20:27.557 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:20:27.557 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:20:27.886 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:20:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:20:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:20:27.888 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:20:27.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:27 smithi184 ceph-mon[100622]: from='client.58149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:27.888 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:27 smithi184 ceph-mon[102844]: from='client.58149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:27 smithi196 ceph-mon[76349]: from='client.58149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:20:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:20:28.161 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:20:28.472 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:28.472 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:28.472 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:28.472 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:28.472 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: from='client.58155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2813248178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: from='client.68135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3470532151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: from='client.58155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2813248178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: from='client.68135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3470532151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: from='client.58155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2813248178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: from='client.68135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3470532151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:30 smithi196 ceph-mon[76349]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:31.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:30 smithi184 ceph-mon[100622]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:31.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:30 smithi184 ceph-mon[102844]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:31.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:31.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:31.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:20:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:32 smithi184 ceph-mon[100622]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:32 smithi184 ceph-mon[102844]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:32 smithi196 ceph-mon[76349]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:34 smithi184 ceph-mon[100622]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:34 smithi184 ceph-mon[102844]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:34 smithi196 ceph-mon[76349]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:36 smithi184 ceph-mon[100622]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:36 smithi184 ceph-mon[102844]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:36 smithi196 ceph-mon[76349]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:38 smithi196 ceph-mon[76349]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:38 smithi184 ceph-mon[100622]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:38 smithi184 ceph-mon[102844]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:40 smithi196 ceph-mon[76349]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:40 smithi184 ceph-mon[100622]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:40 smithi184 ceph-mon[102844]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:41.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:41.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:41.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:20:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[100622]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[102844]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:43 smithi196 ceph-mon[76349]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[100622]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[102844]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:44 smithi196 ceph-mon[76349]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:20:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:47.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:46 smithi184 ceph-mon[100622]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:47.035 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:47.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:46 smithi184 ceph-mon[102844]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:47.035 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:46 smithi196 ceph-mon[76349]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:48.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:48 smithi184 ceph-mon[100622]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:48.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:48 smithi184 ceph-mon[102844]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:48.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:48 smithi196 ceph-mon[76349]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:51.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:50 smithi184 ceph-mon[100622]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:51.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:50 smithi184 ceph-mon[102844]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:50 smithi196 ceph-mon[76349]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:51.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:51.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:20:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:20:51.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:20:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:20:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:20:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:52 smithi184 ceph-mon[100622]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:52 smithi184 ceph-mon[102844]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:52 smithi196 ceph-mon[76349]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:54 smithi184 ceph-mon[100622]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:54 smithi184 ceph-mon[102844]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:54 smithi196 ceph-mon[76349]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:56 smithi184 ceph-mon[100622]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:56 smithi184 ceph-mon[102844]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:56 smithi196 ceph-mon[76349]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:58.772 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:20:58.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:58 smithi184 ceph-mon[100622]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:58.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:58 smithi184 ceph-mon[102844]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:58 smithi196 ceph-mon[76349]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:59.300 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 3h - - 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 15s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 17s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 15s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 17s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 17s ago 3h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 15s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 17s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:20:59.301 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 15s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 17s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 17s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 17s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 17s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 15s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 15s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 15s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 15s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 15s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 17s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:20:59.302 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 15s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:20:59.617 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:20:59.617 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:20:59.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:20:59.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:20:59.618 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:59 smithi196 ceph-mon[76349]: from='client.58185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:59 smithi196 ceph-mon[76349]: from='client.68153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:20:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2688375723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:20:59.895 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:20:59.896 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:20:59.896 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:20:59.896 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:20:59.896 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:20:59.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:20:59.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[100622]: from='client.58185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:59.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[100622]: from='client.68153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2688375723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:59.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[102844]: from='client.58185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:59.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[102844]: from='client.68153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:59.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:20:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2688375723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:00.205 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:21:00.205 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:00.205 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:00.205 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:00.205 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:01.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[100622]: from='client.68159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[100622]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:01.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[100622]: from='client.58209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/86634302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:01.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[102844]: from='client.68159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[102844]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:01.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[102844]: from='client.58209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/86634302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:00 smithi196 ceph-mon[76349]: from='client.68159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:00 smithi196 ceph-mon[76349]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:00 smithi196 ceph-mon[76349]: from='client.58209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/86634302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:01.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:01.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:01.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:02.294 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[100622]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[102844]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:02 smithi196 ceph-mon[76349]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:04 smithi196 ceph-mon[76349]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[100622]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[102844]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:07.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:06 smithi184 ceph-mon[100622]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:07.061 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:07.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:06 smithi184 ceph-mon[102844]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:07.062 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:06 smithi196 ceph-mon[76349]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:08 smithi184 ceph-mon[100622]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:08 smithi184 ceph-mon[102844]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:08 smithi196 ceph-mon[76349]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:10 smithi196 ceph-mon[76349]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:10 smithi184 ceph-mon[100622]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:10 smithi184 ceph-mon[102844]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:11.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:11.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:11.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:21:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:12 smithi196 ceph-mon[76349]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:12 smithi184 ceph-mon[100622]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:12 smithi184 ceph-mon[102844]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:14 smithi196 ceph-mon[76349]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:14 smithi184 ceph-mon[100622]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:14 smithi184 ceph-mon[102844]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:16 smithi196 ceph-mon[76349]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:16 smithi184 ceph-mon[100622]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:16 smithi184 ceph-mon[102844]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:19.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:18 smithi196 ceph-mon[76349]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:18 smithi184 ceph-mon[100622]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:18 smithi184 ceph-mon[102844]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:21.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:20 smithi196 ceph-mon[76349]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:20 smithi184 ceph-mon[100622]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:20 smithi184 ceph-mon[102844]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:21.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:21.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:21.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:21:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:22 smithi196 ceph-mon[76349]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[100622]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[102844]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:23.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[100622]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[102844]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:24 smithi196 ceph-mon[76349]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:26 smithi184 ceph-mon[100622]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:26 smithi184 ceph-mon[102844]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:26 smithi196 ceph-mon[76349]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:28 smithi184 ceph-mon[100622]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:28 smithi184 ceph-mon[102844]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:28 smithi196 ceph-mon[76349]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:30.511 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:21:31.038 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:21:31.038 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 3h - - 2024-09-20T23:21:31.038 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 3h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 8s ago 3h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:21:31.039 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 8s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:21:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:21:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:30 smithi184 ceph-mon[100622]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:30 smithi184 ceph-mon[102844]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:31.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:31.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:31.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:21:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:21:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:21:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:21:31.353 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:21:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:30 smithi196 ceph-mon[76349]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:31.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:21:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:21:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:21:31.621 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:21:31.936 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:21:31.936 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:31.936 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:31.936 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:31.936 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:21:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[100622]: from='client.68183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[100622]: from='client.68189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[100622]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1004058387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3386350884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[102844]: from='client.68183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[102844]: from='client.68189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[102844]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1004058387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3386350884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:31 smithi196 ceph-mon[76349]: from='client.68183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:31 smithi196 ceph-mon[76349]: from='client.68189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:31 smithi196 ceph-mon[76349]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1004058387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3386350884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:32 smithi184 ceph-mon[100622]: from='client.58245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:33.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:32 smithi184 ceph-mon[100622]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:32 smithi184 ceph-mon[102844]: from='client.58245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:33.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:32 smithi184 ceph-mon[102844]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:32 smithi196 ceph-mon[76349]: from='client.58245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:32 smithi196 ceph-mon[76349]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:35.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:34 smithi184 ceph-mon[100622]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:35.303 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:34 smithi184 ceph-mon[102844]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:34 smithi196 ceph-mon[76349]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:36 smithi184 ceph-mon[100622]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:36 smithi184 ceph-mon[102844]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:36 smithi196 ceph-mon[76349]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:38 smithi184 ceph-mon[100622]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:38 smithi184 ceph-mon[102844]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:38 smithi196 ceph-mon[76349]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:41.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:40 smithi196 ceph-mon[76349]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:40 smithi184 ceph-mon[100622]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:40 smithi184 ceph-mon[102844]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:41.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:41.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:41.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:21:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[100622]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[102844]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:42 smithi196 ceph-mon[76349]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:45 smithi196 ceph-mon[76349]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[100622]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[102844]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:21:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:47 smithi196 ceph-mon[76349]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:47 smithi184 ceph-mon[100622]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:47 smithi184 ceph-mon[102844]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:49 smithi196 ceph-mon[76349]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:49.463 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:49 smithi184 ceph-mon[100622]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:49.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:49 smithi184 ceph-mon[102844]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:51 smithi184 ceph-mon[100622]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:51 smithi184 ceph-mon[102844]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:51.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:51.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:21:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:21:51.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:21:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:51 smithi196 ceph-mon[76349]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:21:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:21:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:53 smithi196 ceph-mon[76349]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:53 smithi184 ceph-mon[100622]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:53 smithi184 ceph-mon[102844]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:55 smithi196 ceph-mon[76349]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:55 smithi184 ceph-mon[100622]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:55 smithi184 ceph-mon[102844]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:57 smithi196 ceph-mon[76349]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:57 smithi184 ceph-mon[100622]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:57 smithi184 ceph-mon[102844]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:59 smithi196 ceph-mon[76349]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:59 smithi184 ceph-mon[102844]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:59 smithi184 ceph-mon[100622]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:01 smithi184 ceph-mon[102844]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:01 smithi184 ceph-mon[100622]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:01.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:01.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:01.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:01.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:01 smithi196 ceph-mon[76349]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:02.243 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:22:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:22:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:22:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T23:22:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 3h - - 2024-09-20T23:22:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 18s ago 3h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 3h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 18s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:22:02.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:22:02.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:22:03.103 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:22:03.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:22:03.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:22:03.105 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:22:03.105 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:22:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:22:03.375 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:22:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[100622]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[100622]: from='client.68219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[102844]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:03.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[102844]: from='client.68219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:03.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:03 smithi196 ceph-mon[76349]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:03 smithi196 ceph-mon[76349]: from='client.68219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:22:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:03.688 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:04 smithi196 ceph-mon[76349]: from='client.68225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:04 smithi196 ceph-mon[76349]: from='client.68231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2643993227' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2413873843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[100622]: from='client.68225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[100622]: from='client.68231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2643993227' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2413873843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[102844]: from='client.68225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[102844]: from='client.68231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2643993227' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2413873843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[100622]: from='client.68243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[100622]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[102844]: from='client.68243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[102844]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:05 smithi196 ceph-mon[76349]: from='client.68243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:05 smithi196 ceph-mon[76349]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:07 smithi184 ceph-mon[100622]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:07 smithi184 ceph-mon[102844]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:07 smithi196 ceph-mon[76349]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:09.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:09 smithi184 ceph-mon[100622]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:09.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:09 smithi184 ceph-mon[102844]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:09 smithi196 ceph-mon[76349]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:11.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:11 smithi184 ceph-mon[100622]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:11.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:11 smithi184 ceph-mon[102844]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:11.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:11.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:11.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:11.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:11.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:11.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:11 smithi196 ceph-mon[76349]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:22:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:13 smithi184 ceph-mon[100622]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:13 smithi184 ceph-mon[102844]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:13 smithi196 ceph-mon[76349]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:15 smithi184 ceph-mon[100622]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:15 smithi184 ceph-mon[102844]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:15 smithi196 ceph-mon[76349]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:17 smithi184 ceph-mon[102844]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:17 smithi184 ceph-mon[100622]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:17 smithi196 ceph-mon[76349]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:19 smithi184 ceph-mon[100622]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:19 smithi184 ceph-mon[102844]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:19 smithi196 ceph-mon[76349]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:21 smithi184 ceph-mon[100622]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:21.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:21 smithi184 ceph-mon[102844]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:21.383 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:21.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:21.383 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:21.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:21.384 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:21.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:21.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:21 smithi196 ceph-mon[76349]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:22.282 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:22:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:23 smithi196 ceph-mon[76349]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[100622]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[102844]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[100622]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[102844]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:25 smithi196 ceph-mon[76349]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:27 smithi184 ceph-mon[100622]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:27 smithi184 ceph-mon[102844]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:27 smithi196 ceph-mon[76349]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:29 smithi184 ceph-mon[100622]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:29 smithi184 ceph-mon[102844]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:29 smithi196 ceph-mon[76349]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:31.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:31.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:31.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:31.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:31.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:31 smithi184 ceph-mon[100622]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:31.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:31 smithi184 ceph-mon[102844]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:31 smithi196 ceph-mon[76349]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:22:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:33 smithi184 ceph-mon[100622]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:33.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:33 smithi184 ceph-mon[102844]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:33 smithi196 ceph-mon[76349]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:33.991 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:22:34.539 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 3h - - 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 3h - - 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 11s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 10s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 10s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 3h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 10s ago 3h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 11s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:22:34.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 10s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 10s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 10s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 10s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 10s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:22:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 10s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:34.850 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:22:34.851 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:22:35.120 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:22:35.121 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:22:35.440 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:22:35.440 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:35.440 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:35.440 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:35.440 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:35.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[100622]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[100622]: from='client.68255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[100622]: from='client.58299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/858428422' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[102844]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[102844]: from='client.68255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[102844]: from='client.58299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/858428422' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:35 smithi196 ceph-mon[76349]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:35 smithi196 ceph-mon[76349]: from='client.68255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:35 smithi196 ceph-mon[76349]: from='client.58299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/858428422' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[102844]: from='client.68267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[102844]: from='client.58317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2882529494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[100622]: from='client.68267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[100622]: from='client.58317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2882529494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:36 smithi196 ceph-mon[76349]: from='client.68267 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:36 smithi196 ceph-mon[76349]: from='client.58317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2882529494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:37 smithi184 ceph-mon[100622]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:37 smithi184 ceph-mon[102844]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:37 smithi196 ceph-mon[76349]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:39 smithi184 ceph-mon[100622]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:39 smithi184 ceph-mon[102844]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:39 smithi196 ceph-mon[76349]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:41 smithi184 ceph-mon[100622]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:41 smithi184 ceph-mon[102844]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:41.034Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:41.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:41.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:41.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:41 smithi196 ceph-mon[76349]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:22:43.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:43 smithi196 ceph-mon[76349]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:43.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.367 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[100622]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[102844]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[100622]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[102844]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:45 smithi196 ceph-mon[76349]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:22:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:46 smithi184 ceph-mon[100622]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:46 smithi184 ceph-mon[102844]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:46 smithi196 ceph-mon[76349]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:48 smithi184 ceph-mon[100622]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:48 smithi184 ceph-mon[102844]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:48 smithi196 ceph-mon[76349]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:51.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:50 smithi184 ceph-mon[100622]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:51.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:50 smithi184 ceph-mon[102844]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:50 smithi196 ceph-mon[76349]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:51.296 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:51.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:51.296 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:51.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:22:51.296 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:22:51.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:22:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:22:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:22:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:52 smithi184 ceph-mon[102844]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:53.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:52 smithi184 ceph-mon[100622]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:52 smithi196 ceph-mon[76349]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:54 smithi184 ceph-mon[102844]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:55.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:54 smithi184 ceph-mon[100622]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:54 smithi196 ceph-mon[76349]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:56 smithi184 ceph-mon[100622]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:56 smithi184 ceph-mon[102844]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:56 smithi196 ceph-mon[76349]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:58 smithi184 ceph-mon[100622]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:58 smithi184 ceph-mon[102844]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:58 smithi196 ceph-mon[76349]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:01.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:00 smithi184 ceph-mon[100622]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:01.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:00 smithi184 ceph-mon[102844]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:00 smithi196 ceph-mon[76349]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:01.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:01.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:01.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:23:03.015 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:02 smithi196 ceph-mon[76349]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:02 smithi184 ceph-mon[100622]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:02 smithi184 ceph-mon[102844]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:04.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:05 smithi196 ceph-mon[76349]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[100622]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[102844]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:23:06.263 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:23:06.263 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 3h - - 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 1s ago 3h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 3s ago 3h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 3s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 1s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:23:06.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 3s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 3s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 3s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 3s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:23:06.265 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:23:06.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:23:06.266 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:23:06.573 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:23:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:23:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:23:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:23:06.574 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:23:06.842 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:23:06.843 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:23:06.843 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:23:06.843 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:23:06.843 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:23:07.153 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:23:07.153 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:07.153 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:07.154 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:07.154 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: from='client.68291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: from='client.68303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1886391651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:07.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: from='client.68291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: from='client.68303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1886391651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: from='client.68291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: from='client.68303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1886391651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:08 smithi196 ceph-mon[76349]: from='client.58353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2929191654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:08 smithi184 ceph-mon[100622]: from='client.58353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2929191654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:08 smithi184 ceph-mon[102844]: from='client.58353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2929191654' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:09 smithi196 ceph-mon[76349]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:09 smithi184 ceph-mon[100622]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:09 smithi184 ceph-mon[102844]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:11.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:11 smithi184 ceph-mon[100622]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:11.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:11 smithi184 ceph-mon[102844]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:11.214 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:11.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:11.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:11.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:11.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:11.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:11 smithi196 ceph-mon[76349]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:12.243 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:23:13.351 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:13 smithi184 ceph-mon[100622]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:13.352 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:13 smithi184 ceph-mon[102844]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:13 smithi196 ceph-mon[76349]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:15.363 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:15 smithi196 ceph-mon[76349]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:15 smithi184 ceph-mon[100622]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:15 smithi184 ceph-mon[102844]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:17 smithi196 ceph-mon[76349]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:17 smithi184 ceph-mon[100622]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:17 smithi184 ceph-mon[102844]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:19 smithi196 ceph-mon[76349]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:19 smithi184 ceph-mon[100622]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:19 smithi184 ceph-mon[102844]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:21 smithi184 ceph-mon[100622]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:21 smithi184 ceph-mon[102844]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:21.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:21.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:21.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:21 smithi196 ceph-mon[76349]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:23:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:23 smithi196 ceph-mon[76349]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:23 smithi184 ceph-mon[100622]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:23 smithi184 ceph-mon[102844]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:25 smithi196 ceph-mon[76349]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[100622]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[102844]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:27 smithi184 ceph-mon[100622]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:27 smithi184 ceph-mon[102844]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:27 smithi196 ceph-mon[76349]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:29 smithi184 ceph-mon[100622]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:29 smithi184 ceph-mon[102844]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:29 smithi196 ceph-mon[76349]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:31 smithi184 ceph-mon[100622]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:31 smithi184 ceph-mon[102844]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:31.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:31.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:31.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:31 smithi196 ceph-mon[76349]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:23:33.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:33 smithi184 ceph-mon[100622]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:33.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:33 smithi184 ceph-mon[102844]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:33 smithi196 ceph-mon[76349]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:35 smithi184 ceph-mon[100622]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:35 smithi184 ceph-mon[102844]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:35 smithi196 ceph-mon[76349]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:37.452 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:23:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:37 smithi184 ceph-mon[100622]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:37 smithi184 ceph-mon[102844]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:37 smithi196 ceph-mon[76349]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 3h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 13s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:23:37.982 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 13s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 13s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 13s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 13s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 3h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:23:37.983 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 13s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:23:38.295 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:23:38.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:23:38.297 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:23:38.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:23:38.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:23:38.297 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:23:38.564 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:23:38.565 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:23:38.872 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:23:38.873 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:38.873 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:38.873 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:38.873 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: from='client.58365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: from='client.58371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: from='client.68339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/496424596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3294808198' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: from='client.58365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: from='client.58371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: from='client.68339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/496424596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3294808198' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: from='client.58365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: from='client.58371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: from='client.68339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/496424596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3294808198' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:40 smithi184 ceph-mon[100622]: from='client.68351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:40 smithi184 ceph-mon[102844]: from='client.68351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:40 smithi196 ceph-mon[76349]: from='client.68351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:41 smithi184 ceph-mon[100622]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:41 smithi184 ceph-mon[102844]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:41.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:41.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:41.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:41 smithi196 ceph-mon[76349]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:23:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:43 smithi196 ceph-mon[76349]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:43 smithi184 ceph-mon[100622]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:43 smithi184 ceph-mon[102844]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:44.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:44 smithi196 ceph-mon[76349]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:44.453 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:44.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.588 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[100622]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[102844]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:44.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:46.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:23:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:46 smithi196 ceph-mon[76349]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:46 smithi184 ceph-mon[100622]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:46 smithi184 ceph-mon[102844]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:48 smithi184 ceph-mon[100622]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:48 smithi184 ceph-mon[102844]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:48 smithi196 ceph-mon[76349]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:50 smithi184 ceph-mon[100622]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:50 smithi184 ceph-mon[102844]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:51.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:51.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:23:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:23:51.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:23:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:50 smithi196 ceph-mon[76349]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:52.259 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:23:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:23:53.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:52 smithi184 ceph-mon[100622]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:53.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:52 smithi184 ceph-mon[102844]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:52 smithi196 ceph-mon[76349]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:54 smithi184 ceph-mon[100622]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:54 smithi184 ceph-mon[102844]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:54 smithi196 ceph-mon[76349]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:56 smithi184 ceph-mon[100622]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:56 smithi184 ceph-mon[102844]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:56 smithi196 ceph-mon[76349]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:58 smithi184 ceph-mon[100622]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:58 smithi184 ceph-mon[102844]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:58 smithi196 ceph-mon[76349]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:00 smithi184 ceph-mon[100622]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:00 smithi184 ceph-mon[102844]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:01.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:01.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:01.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:00 smithi196 ceph-mon[76349]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:24:03.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:02 smithi184 ceph-mon[100622]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:03.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:02 smithi184 ceph-mon[102844]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:02 smithi196 ceph-mon[76349]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:04.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:04 smithi196 ceph-mon[76349]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[100622]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:04 smithi184 ceph-mon[102844]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:06.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:06 smithi184 ceph-mon[100622]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:24:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:06 smithi184 ceph-mon[102844]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:24:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:06 smithi196 ceph-mon[76349]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:24:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:09.170 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:24:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:08 smithi184 ceph-mon[100622]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:08 smithi184 ceph-mon[102844]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:08 smithi196 ceph-mon[76349]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:09.709 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:24:09.709 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 3h - - 2024-09-20T23:24:09.709 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 6s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 6s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 6s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 6s ago 3h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 6s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 6s ago 3h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 6s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 6s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 6s ago 3h 71.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:24:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 6s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 3h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 3h 72.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 6s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:24:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:24:10.054 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:24:10.054 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:24:10.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:24:10.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:24:10.055 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:24:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:10 smithi184 ceph-mon[100622]: from='client.68363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:10 smithi184 ceph-mon[102844]: from='client.68363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:10.321 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:24:10.321 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:24:10.322 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:24:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:10 smithi196 ceph-mon[76349]: from='client.68363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:10.631 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:24:10.632 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:10.632 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:10.632 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[100622]: from='client.68369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[100622]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:24:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[100622]: from='client.68375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1922830075' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3087777941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[102844]: from='client.68369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[102844]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[102844]: from='client.68375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1922830075' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3087777941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:11.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:11.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:11.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:11 smithi196 ceph-mon[76349]: from='client.68369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:11 smithi196 ceph-mon[76349]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:11 smithi196 ceph-mon[76349]: from='client.68375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1922830075' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3087777941' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:12 smithi184 ceph-mon[100622]: from='client.58425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:12 smithi184 ceph-mon[102844]: from='client.58425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:24:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:12 smithi196 ceph-mon[76349]: from='client.58425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:13 smithi184 ceph-mon[100622]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:24:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:13 smithi184 ceph-mon[102844]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:24:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:13 smithi196 ceph-mon[76349]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-20T23:24:14.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:14.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:15.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:15 smithi184 ceph-mon[100622]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:24:15.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:15 smithi184 ceph-mon[102844]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:24:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:15 smithi196 ceph-mon[76349]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:24:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:17 smithi184 ceph-mon[100622]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:24:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:17 smithi184 ceph-mon[102844]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:24:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:17 smithi196 ceph-mon[76349]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:24:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:19 smithi184 ceph-mon[100622]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:24:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:19 smithi184 ceph-mon[102844]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:24:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:19 smithi196 ceph-mon[76349]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:24:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:21 smithi184 ceph-mon[100622]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:24:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:21 smithi184 ceph-mon[102844]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:24:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:21.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:21.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:21.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:21 smithi196 ceph-mon[76349]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:24:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:24:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:23 smithi184 ceph-mon[100622]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:24:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:23 smithi184 ceph-mon[102844]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:24:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:23 smithi196 ceph-mon[76349]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:24:24.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:24.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:24.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:24.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:24 smithi196 ceph-mon[76349]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[100622]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:24 smithi184 ceph-mon[102844]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:27 smithi196 ceph-mon[76349]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:24:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:27 smithi184 ceph-mon[100622]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:24:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:27 smithi184 ceph-mon[102844]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T23:24:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:29 smithi196 ceph-mon[76349]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:24:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:29 smithi184 ceph-mon[100622]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:24:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:29 smithi184 ceph-mon[102844]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:24:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:31 smithi184 ceph-mon[102844]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:31 smithi184 ceph-mon[100622]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:31.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:31.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:31.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:31 smithi196 ceph-mon[76349]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:24:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:24:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:33 smithi196 ceph-mon[76349]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-20T23:24:33.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:33 smithi184 ceph-mon[100622]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-20T23:24:33.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:33 smithi184 ceph-mon[102844]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-20T23:24:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:35 smithi196 ceph-mon[76349]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:24:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:35 smithi184 ceph-mon[100622]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:24:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:35 smithi184 ceph-mon[102844]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:24:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:37 smithi184 ceph-mon[100622]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T23:24:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:37 smithi184 ceph-mon[102844]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T23:24:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:37 smithi196 ceph-mon[76349]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T23:24:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:39 smithi184 ceph-mon[100622]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:24:39.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:39 smithi184 ceph-mon[102844]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:24:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:39 smithi196 ceph-mon[76349]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:24:40.929 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:24:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:41 smithi184 ceph-mon[100622]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T23:24:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:41 smithi184 ceph-mon[102844]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T23:24:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:41.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:41.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:41.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:24:41.458 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 17s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 17s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 17s ago 3h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 3h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 17s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 17s ago 3h 79.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 17s ago 3h 81.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 17s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 17s ago 3h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 16s ago 3h 86.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 16s ago 3h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 16s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:24:41.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 16s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:24:41.460 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 16s ago 3h 72.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:24:41.460 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:24:41.460 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:24:41.460 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 17s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:24:41.460 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 16s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:24:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:41 smithi196 ceph-mon[76349]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:41.769 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:24:41.770 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:24:42.040 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:24:42.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:24:42.042 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:24:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[102844]: from='client.68399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[102844]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/18771718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[100622]: from='client.68399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[100622]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/18771718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:42.355 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:24:42.355 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:42.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:42.355 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:42.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:42 smithi196 ceph-mon[76349]: from='client.68399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:42 smithi196 ceph-mon[76349]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/18771718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[100622]: from='client.68411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[100622]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[100622]: from='client.58461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2749473697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[102844]: from='client.68411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[102844]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[102844]: from='client.58461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2749473697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:43 smithi196 ceph-mon[76349]: from='client.68411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:43 smithi196 ceph-mon[76349]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:24:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:43 smithi196 ceph-mon[76349]: from='client.58461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2749473697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[100622]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[102844]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:44 smithi196 ceph-mon[76349]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:24:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:46.504 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:24:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:47 smithi196 ceph-mon[76349]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:47 smithi184 ceph-mon[100622]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:47 smithi184 ceph-mon[102844]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:24:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:49.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:49 smithi196 ceph-mon[76349]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:49 smithi184 ceph-mon[100622]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:49 smithi184 ceph-mon[102844]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:51 smithi184 ceph-mon[100622]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:51 smithi184 ceph-mon[102844]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:51.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:51.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:24:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:24:51.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:24:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:51 smithi196 ceph-mon[76349]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:24:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:24:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:53 smithi184 ceph-mon[100622]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:53 smithi184 ceph-mon[102844]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:53 smithi196 ceph-mon[76349]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:55.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:55 smithi184 ceph-mon[100622]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:55.412 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:55 smithi184 ceph-mon[102844]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:55 smithi196 ceph-mon[76349]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:57 smithi184 ceph-mon[100622]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:57 smithi184 ceph-mon[102844]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:57 smithi196 ceph-mon[76349]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:59 smithi184 ceph-mon[100622]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:59 smithi184 ceph-mon[102844]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:59 smithi196 ceph-mon[76349]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:01 smithi184 ceph-mon[102844]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:01.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:01 smithi184 ceph-mon[100622]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:01.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:01.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:01.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:01 smithi196 ceph-mon[76349]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:25:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:03 smithi184 ceph-mon[100622]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:03 smithi184 ceph-mon[102844]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:03 smithi196 ceph-mon[76349]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:04 smithi196 ceph-mon[76349]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[100622]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[102844]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:05.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:06 smithi196 ceph-mon[76349]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:06.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[102844]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[100622]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:08 smithi184 ceph-mon[100622]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:08 smithi184 ceph-mon[102844]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:08 smithi196 ceph-mon[76349]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:11.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:10 smithi184 ceph-mon[100622]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:11.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:10 smithi184 ceph-mon[102844]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:10 smithi196 ceph-mon[76349]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:11.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:11.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:11.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:25:12.648 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:25:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:12 smithi184 ceph-mon[100622]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:12 smithi184 ceph-mon[102844]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:12 smithi196 ceph-mon[76349]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:13.179 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:25:13.179 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T23:25:13.179 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T23:25:13.179 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 9s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 9s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 9s ago 3h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 9s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 9s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 9s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 9s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:25:13.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 3h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 3h 73.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:25:13.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:25:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:25:13.490 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:25:13.763 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:25:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:25:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:25:13.763 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:25:13.764 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:25:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[102844]: from='client.68435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[102844]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[102844]: from='client.68447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3548002349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[100622]: from='client.68435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[100622]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[100622]: from='client.68447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3548002349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:14.076 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:25:14.076 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:14.076 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:13 smithi196 ceph-mon[76349]: from='client.68435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:13 smithi196 ceph-mon[76349]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:13 smithi196 ceph-mon[76349]: from='client.68447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3548002349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[100622]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[100622]: from='client.58497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3038081807' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[102844]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[102844]: from='client.58497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:14.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:14.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3038081807' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:14 smithi196 ceph-mon[76349]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:14 smithi196 ceph-mon[76349]: from='client.58497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:15.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3038081807' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:16 smithi184 ceph-mon[100622]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:16 smithi184 ceph-mon[102844]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:16 smithi196 ceph-mon[76349]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:18 smithi184 ceph-mon[100622]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:18 smithi184 ceph-mon[102844]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:18 smithi196 ceph-mon[76349]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:21.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:20 smithi184 ceph-mon[100622]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:20 smithi184 ceph-mon[102844]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:20 smithi196 ceph-mon[76349]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:21.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:21.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:21.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:25:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:22 smithi184 ceph-mon[100622]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:22 smithi184 ceph-mon[102844]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:22 smithi196 ceph-mon[76349]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[102844]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[100622]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:24 smithi196 ceph-mon[76349]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:26 smithi196 ceph-mon[76349]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:26.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[100622]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[102844]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:29.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:28 smithi196 ceph-mon[76349]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:28 smithi184 ceph-mon[100622]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:28 smithi184 ceph-mon[102844]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:30 smithi184 ceph-mon[100622]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:30 smithi184 ceph-mon[102844]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:30 smithi196 ceph-mon[76349]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:31.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:31.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:31.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:25:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:32 smithi184 ceph-mon[100622]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:32 smithi184 ceph-mon[102844]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:32 smithi196 ceph-mon[76349]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:34 smithi184 ceph-mon[100622]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:34 smithi184 ceph-mon[102844]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:34 smithi196 ceph-mon[76349]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:36.845 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:36 smithi184 ceph-mon[102844]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:36.845 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:36.845 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:36 smithi184 ceph-mon[100622]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:36.846 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:36 smithi196 ceph-mon[76349]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:38 smithi184 ceph-mon[100622]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:38 smithi184 ceph-mon[102844]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:38 smithi196 ceph-mon[76349]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:40 smithi184 ceph-mon[100622]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:40 smithi184 ceph-mon[102844]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:40 smithi196 ceph-mon[76349]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:41.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:41.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:41.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:25:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:42 smithi184 ceph-mon[100622]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:42 smithi184 ceph-mon[102844]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:42 smithi196 ceph-mon[76349]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:25:44.894 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 3h 62.8M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 3h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 19s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:25:44.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 19s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 19s ago 3h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 19s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 19s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 19s ago 3h 73.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:25:44.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 19s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[100622]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[102844]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:44 smithi196 ceph-mon[76349]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:25:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:25:45.207 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:25:45.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:25:45.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:25:45.207 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:25:45.474 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:25:45.475 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:25:45.793 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:25:45.793 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:45.793 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:45.793 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:45.793 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='client.68471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='client.68477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='client.58521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3703516054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/645861336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='client.68471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='client.68477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='client.58521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3703516054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/645861336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='client.68471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='client.68477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='client.58521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3703516054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:25:46.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/645861336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:46 smithi196 ceph-mon[76349]: from='client.68495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:46 smithi196 ceph-mon[76349]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[100622]: from='client.68495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[100622]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[102844]: from='client.68495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[102844]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:48 smithi196 ceph-mon[76349]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:48 smithi184 ceph-mon[100622]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:48 smithi184 ceph-mon[102844]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:50 smithi196 ceph-mon[76349]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:50 smithi184 ceph-mon[100622]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:50 smithi184 ceph-mon[102844]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:51.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:51.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:25:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:25:51.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:25:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:25:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:25:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:52 smithi196 ceph-mon[76349]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:52 smithi184 ceph-mon[100622]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:52 smithi184 ceph-mon[102844]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:54 smithi196 ceph-mon[76349]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:54 smithi184 ceph-mon[100622]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:54 smithi184 ceph-mon[102844]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:56 smithi184 ceph-mon[100622]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:57.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:56 smithi184 ceph-mon[102844]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:56 smithi196 ceph-mon[76349]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:58 smithi196 ceph-mon[76349]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:58 smithi184 ceph-mon[100622]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:58 smithi184 ceph-mon[102844]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:00 smithi196 ceph-mon[76349]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:00 smithi184 ceph-mon[100622]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:00 smithi184 ceph-mon[102844]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:01.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:01.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:01.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:02] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:26:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:02 smithi196 ceph-mon[76349]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:02 smithi184 ceph-mon[100622]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:02 smithi184 ceph-mon[102844]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[100622]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[102844]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:05 smithi196 ceph-mon[76349]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[100622]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[102844]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:06 smithi196 ceph-mon[76349]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:08 smithi184 ceph-mon[100622]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:08 smithi184 ceph-mon[102844]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:09.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:08 smithi196 ceph-mon[76349]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:10 smithi184 ceph-mon[100622]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:10 smithi184 ceph-mon[102844]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:10 smithi196 ceph-mon[76349]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:11.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:11.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:11.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:12] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:26:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:12 smithi184 ceph-mon[100622]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:12 smithi184 ceph-mon[102844]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:12 smithi196 ceph-mon[76349]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:14 smithi184 ceph-mon[102844]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:14 smithi184 ceph-mon[100622]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:14 smithi196 ceph-mon[76349]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:16.090 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 3h - - 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 3h - - 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 12s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 10s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:26:16.633 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 12s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 10s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 12s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 12s ago 3h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 10s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 12s ago 3h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 12s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 10s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 12s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 12s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 12s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 12s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 10s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 10s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:26:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 10s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 10s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 3h 68.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 12s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:26:16.635 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 10s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:26:16.952 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:26:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[102844]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[102844]: from='client.58545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[100622]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[100622]: from='client.58545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:16 smithi196 ceph-mon[76349]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:16 smithi196 ceph-mon[76349]: from='client.58545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:26:17.230 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:26:17.231 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:26:17.542 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:26:17.542 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:17.542 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:17.542 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:17.542 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[102844]: from='client.58551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[102844]: from='client.68519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3894144398' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[102844]: from='client.68531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3881048741' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[100622]: from='client.58551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[100622]: from='client.68519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3894144398' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[100622]: from='client.68531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3881048741' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:17 smithi196 ceph-mon[76349]: from='client.58551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:17 smithi196 ceph-mon[76349]: from='client.68519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3894144398' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:17 smithi196 ceph-mon[76349]: from='client.68531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3881048741' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:19.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:18 smithi196 ceph-mon[76349]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:18 smithi184 ceph-mon[102844]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:18 smithi184 ceph-mon[100622]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:20 smithi184 ceph-mon[100622]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:20 smithi184 ceph-mon[102844]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:20 smithi196 ceph-mon[76349]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:21.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:21.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:21.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:26:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:22 smithi184 ceph-mon[100622]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:22 smithi184 ceph-mon[102844]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:22 smithi196 ceph-mon[76349]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:24 smithi196 ceph-mon[76349]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[100622]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[102844]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[100622]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[102844]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:26 smithi196 ceph-mon[76349]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:28 smithi196 ceph-mon[76349]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:28 smithi184 ceph-mon[100622]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:28 smithi184 ceph-mon[102844]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:30 smithi184 ceph-mon[102844]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:31.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:30 smithi184 ceph-mon[100622]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:31.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:31.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:31.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:31.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:30 smithi196 ceph-mon[76349]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:26:33.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:32 smithi184 ceph-mon[100622]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:33.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:32 smithi184 ceph-mon[102844]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:32 smithi196 ceph-mon[76349]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:34 smithi184 ceph-mon[100622]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:34 smithi184 ceph-mon[102844]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:34 smithi196 ceph-mon[76349]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:37.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:36 smithi184 ceph-mon[102844]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:37.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:37.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:36 smithi184 ceph-mon[100622]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:37.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:36 smithi196 ceph-mon[76349]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:39.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:38 smithi184 ceph-mon[100622]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:39.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:38 smithi184 ceph-mon[102844]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:38 smithi196 ceph-mon[76349]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:40 smithi184 ceph-mon[100622]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:40 smithi184 ceph-mon[102844]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:41.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:41.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:41.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:40 smithi196 ceph-mon[76349]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:26:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:42 smithi184 ceph-mon[100622]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:42 smithi184 ceph-mon[102844]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:42 smithi196 ceph-mon[76349]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[102844]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[100622]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:44 smithi196 ceph-mon[76349]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:45.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[100622]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[102844]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:47 smithi196 ceph-mon[76349]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:26:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:47.839 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 3h - - 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:26:48.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 3s ago 3h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 2s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 3s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 3s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 3s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 3s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 3s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 2s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 2s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 2s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 2s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:26:48.370 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:26:48.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:26:48.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:26:48.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:26:48.371 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 2s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:48.681 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:26:48.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:26:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:26:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:26:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:26:48.954 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:26:49.264 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:26:49.264 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:49.264 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:49.264 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:49.264 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:49.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[102844]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:49.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[102844]: from='client.58581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[102844]: from='client.58587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1815934199' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:49.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[100622]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:49.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[100622]: from='client.58581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[100622]: from='client.58587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1815934199' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:49 smithi196 ceph-mon[76349]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:49 smithi196 ceph-mon[76349]: from='client.58581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:49 smithi196 ceph-mon[76349]: from='client.58587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:49.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1815934199' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[100622]: from='client.68555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[100622]: from='client.68567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1017350285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[102844]: from='client.68555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[102844]: from='client.68567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1017350285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:50.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:50 smithi196 ceph-mon[76349]: from='client.68555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:50 smithi196 ceph-mon[76349]: from='client.68567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:50.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1017350285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:51 smithi184 ceph-mon[100622]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:51 smithi184 ceph-mon[102844]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:51.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:51.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:26:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:26:51.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:26:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:51 smithi196 ceph-mon[76349]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:26:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:26:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:53 smithi184 ceph-mon[100622]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:53.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:53 smithi184 ceph-mon[102844]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:53 smithi196 ceph-mon[76349]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:55 smithi184 ceph-mon[100622]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:55 smithi184 ceph-mon[102844]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:55 smithi196 ceph-mon[76349]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:57 smithi196 ceph-mon[76349]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:57 smithi184 ceph-mon[100622]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:57 smithi184 ceph-mon[102844]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:59.322 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:59 smithi184 ceph-mon[100622]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:59.322 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:59.322 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:59 smithi184 ceph-mon[102844]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:59.322 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:59 smithi196 ceph-mon[76349]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:01 smithi196 ceph-mon[76349]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:01 smithi184 ceph-mon[100622]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:01 smithi184 ceph-mon[102844]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:01.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:01.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:01.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:27:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:03 smithi196 ceph-mon[76349]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:03 smithi184 ceph-mon[100622]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:03 smithi184 ceph-mon[102844]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:05 smithi196 ceph-mon[76349]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[102844]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[100622]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:05.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[100622]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[102844]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:07 smithi196 ceph-mon[76349]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:07.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:09.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:09 smithi196 ceph-mon[76349]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:09 smithi184 ceph-mon[100622]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:09 smithi184 ceph-mon[102844]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:11.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:11 smithi196 ceph-mon[76349]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:11 smithi184 ceph-mon[100622]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:11 smithi184 ceph-mon[102844]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:11.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:11.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:11.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:27:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:13 smithi184 ceph-mon[100622]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:13 smithi184 ceph-mon[102844]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:13 smithi196 ceph-mon[76349]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:15 smithi184 ceph-mon[100622]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:15 smithi184 ceph-mon[102844]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:15 smithi196 ceph-mon[76349]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:17 smithi184 ceph-mon[100622]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:17 smithi184 ceph-mon[102844]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:17 smithi196 ceph-mon[76349]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:19.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:19 smithi196 ceph-mon[76349]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:19 smithi184 ceph-mon[100622]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:19 smithi184 ceph-mon[102844]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:19.561 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:27:20.098 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:27:20.098 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 3h - - 2024-09-20T23:27:20.098 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T23:27:20.098 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 15s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:27:20.098 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 15s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 15s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 15s ago 3h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 13s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 15s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 15s ago 3h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 15s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 15s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 15s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 15s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 13s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:27:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 13s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 13s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 13s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 3h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 15s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:27:20.100 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 13s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:20.433 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:27:20.434 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:27:20.714 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:27:20.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:27:21.045 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:27:21.045 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:21.045 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:21.045 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:21.045 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:21.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:21.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:21.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:21.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:21.190 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:21.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: from='client.58617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: from='client.68591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2462641048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1883210389' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: from='client.58617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: from='client.68591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2462641048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1883210389' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: from='client.58617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: from='client.68591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2462641048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1883210389' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:22 smithi184 ceph-mon[100622]: from='client.68603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:22 smithi184 ceph-mon[102844]: from='client.68603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:27:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:22 smithi196 ceph-mon[76349]: from='client.68603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:23 smithi184 ceph-mon[100622]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:23 smithi184 ceph-mon[102844]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:23 smithi196 ceph-mon[76349]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[100622]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[102844]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:25 smithi196 ceph-mon[76349]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[100622]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[102844]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:27 smithi196 ceph-mon[76349]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:29 smithi184 ceph-mon[100622]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:29 smithi184 ceph-mon[102844]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:29 smithi196 ceph-mon[76349]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:31.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:31 smithi196 ceph-mon[76349]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:31 smithi184 ceph-mon[100622]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:31 smithi184 ceph-mon[102844]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:31.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:31.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:31.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:27:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:33 smithi196 ceph-mon[76349]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:33.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:33 smithi184 ceph-mon[100622]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:33.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:33 smithi184 ceph-mon[102844]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:35 smithi196 ceph-mon[76349]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:35 smithi184 ceph-mon[100622]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:35 smithi184 ceph-mon[102844]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:37 smithi196 ceph-mon[76349]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:37 smithi184 ceph-mon[100622]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:37 smithi184 ceph-mon[102844]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:39 smithi196 ceph-mon[76349]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:39.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:39 smithi184 ceph-mon[100622]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:39.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:39 smithi184 ceph-mon[102844]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:41.311 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:41.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:41.311 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:41.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:41.311 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:41.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:41.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:41 smithi184 ceph-mon[100622]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:41.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:41 smithi184 ceph-mon[102844]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:41 smithi196 ceph-mon[76349]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:27:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:43 smithi196 ceph-mon[76349]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:43 smithi184 ceph-mon[100622]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:43 smithi184 ceph-mon[102844]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:45 smithi196 ceph-mon[76349]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[100622]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[102844]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:47 smithi196 ceph-mon[76349]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[100622]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[102844]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:27:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:48 smithi196 ceph-mon[76349]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:48 smithi184 ceph-mon[100622]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:48 smithi184 ceph-mon[102844]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:51.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:50 smithi196 ceph-mon[76349]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:50 smithi184 ceph-mon[100622]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:50 smithi184 ceph-mon[102844]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:51.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:51.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:27:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:27:51.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:27:51.336 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 3h - - 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 3h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:27:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 3h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 5s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 5s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 5s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 5s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:27:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 5s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:27:52.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:27:52.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:27:52.194 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:27:52.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:27:52.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:27:52.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:27:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:27:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:27:52.464 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:27:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:27:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:27:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:27:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:27:52.465 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:27:52.772 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:27:52.772 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:52.772 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:52.772 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: from='client.68615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: from='client.58659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: from='client.68627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1873999591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:52.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2225401760' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: from='client.68615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: from='client.58659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: from='client.68627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1873999591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2225401760' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: from='client.68615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: from='client.58659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: from='client.68627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1873999591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2225401760' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:53 smithi196 ceph-mon[76349]: from='client.58677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:53 smithi184 ceph-mon[100622]: from='client.58677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:53 smithi184 ceph-mon[102844]: from='client.58677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:54 smithi196 ceph-mon[76349]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:54 smithi184 ceph-mon[100622]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:54 smithi184 ceph-mon[102844]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:56 smithi196 ceph-mon[76349]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:56 smithi184 ceph-mon[100622]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:56 smithi184 ceph-mon[102844]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:58 smithi196 ceph-mon[76349]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:58 smithi184 ceph-mon[100622]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:58 smithi184 ceph-mon[102844]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:00 smithi184 ceph-mon[100622]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:00 smithi184 ceph-mon[102844]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:00 smithi196 ceph-mon[76349]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:01.472 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:01.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:01.473 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:01.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:01.473 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:01.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:02.265 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:28:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:02 smithi196 ceph-mon[76349]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:02 smithi184 ceph-mon[100622]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:02 smithi184 ceph-mon[102844]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:04 smithi196 ceph-mon[76349]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:04 smithi184 ceph-mon[100622]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:04 smithi184 ceph-mon[102844]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:06.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[100622]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[102844]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:06 smithi196 ceph-mon[76349]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:07.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:08 smithi184 ceph-mon[100622]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:08 smithi184 ceph-mon[102844]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:08 smithi196 ceph-mon[76349]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:10 smithi184 ceph-mon[100622]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:10 smithi184 ceph-mon[102844]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:11.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:11.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:11.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:11.357 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:10 smithi196 ceph-mon[76349]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:28:13.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:13 smithi196 ceph-mon[76349]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:13 smithi184 ceph-mon[100622]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:13 smithi184 ceph-mon[102844]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:13.956 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:28:13.813021843Z level=info msg="Completed cleanup jobs" duration=12.981657ms 2024-09-20T23:28:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:15 smithi184 ceph-mon[100622]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:15 smithi184 ceph-mon[102844]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:15 smithi196 ceph-mon[76349]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:17 smithi184 ceph-mon[100622]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:17 smithi184 ceph-mon[102844]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:17 smithi196 ceph-mon[76349]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:19 smithi184 ceph-mon[100622]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:19 smithi184 ceph-mon[102844]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:19 smithi196 ceph-mon[76349]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:21 smithi184 ceph-mon[100622]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:21 smithi184 ceph-mon[102844]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:21.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:21.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:21.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:21 smithi196 ceph-mon[76349]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:28:23.065 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:28:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:23 smithi184 ceph-mon[100622]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:23 smithi184 ceph-mon[102844]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:23.364 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:23 smithi196 ceph-mon[76349]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:23.604 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:28:23.604 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T23:28:23.604 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 18s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 18s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 18s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 3h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:28:23.605 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:28:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 3h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:28:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 18s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:28:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:28:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 18s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:28:23.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 18s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 18s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:28:23.607 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:28:23.608 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:28:23.608 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:28:23.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:28:23.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:28:23.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:28:24.200 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:28:24.201 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:28:24.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:24 smithi184 ceph-mon[100622]: from='client.68651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:24.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4002388895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:24.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:24 smithi184 ceph-mon[102844]: from='client.68651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:24.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4002388895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:24 smithi196 ceph-mon[76349]: from='client.68651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4002388895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:24.522 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:28:24.522 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:24.522 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:24.522 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:24.522 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[100622]: from='client.58695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[100622]: from='client.58701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[100622]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[100622]: from='client.58713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/965892569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[102844]: from='client.58695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[102844]: from='client.58701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[102844]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[102844]: from='client.58713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/965892569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:25 smithi196 ceph-mon[76349]: from='client.58695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:25 smithi196 ceph-mon[76349]: from='client.58701 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:25 smithi196 ceph-mon[76349]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:25 smithi196 ceph-mon[76349]: from='client.58713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/965892569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:27 smithi196 ceph-mon[76349]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[100622]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[102844]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:29 smithi196 ceph-mon[76349]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:29 smithi184 ceph-mon[100622]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:29 smithi184 ceph-mon[102844]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:31 smithi196 ceph-mon[76349]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:31 smithi184 ceph-mon[100622]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:31 smithi184 ceph-mon[102844]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:31.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:31.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:31.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:28:33.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:33 smithi196 ceph-mon[76349]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:33 smithi184 ceph-mon[100622]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:33 smithi184 ceph-mon[102844]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:35.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:35 smithi196 ceph-mon[76349]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:35 smithi184 ceph-mon[100622]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:35 smithi184 ceph-mon[102844]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:37 smithi196 ceph-mon[76349]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:37 smithi184 ceph-mon[100622]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:37.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:37 smithi184 ceph-mon[102844]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:39 smithi196 ceph-mon[76349]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:39 smithi184 ceph-mon[100622]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:39 smithi184 ceph-mon[102844]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:41 smithi196 ceph-mon[76349]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:41 smithi184 ceph-mon[100622]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:41 smithi184 ceph-mon[102844]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:41.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:41.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:41.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:42.283 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:28:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:43 smithi184 ceph-mon[100622]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:43 smithi184 ceph-mon[102844]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:43 smithi196 ceph-mon[76349]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:44.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:44.258 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:45 smithi184 ceph-mon[100622]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:45 smithi184 ceph-mon[102844]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:45.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:45 smithi196 ceph-mon[76349]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[100622]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[102844]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:47 smithi196 ceph-mon[76349]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:28:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:49 smithi184 ceph-mon[100622]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:49 smithi184 ceph-mon[102844]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:49 smithi196 ceph-mon[76349]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:51 smithi184 ceph-mon[100622]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:51 smithi184 ceph-mon[102844]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:51.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:51.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:28:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:28:51.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:28:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:51 smithi196 ceph-mon[76349]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:28:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:28:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:53 smithi184 ceph-mon[100622]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:53 smithi184 ceph-mon[102844]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:53.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:53 smithi196 ceph-mon[76349]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:54.817 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 8s ago 3h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:28:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 8s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 3h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 3h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 8s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 8s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 8s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 8s ago 3h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:28:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:28:55.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:28:55.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:28:55.349 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 8s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:28:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:55 smithi196 ceph-mon[76349]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:55 smithi184 ceph-mon[100622]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:55.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:55 smithi184 ceph-mon[102844]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:28:55.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:28:55.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:28:55.663 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:28:55.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:28:55.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:28:55.663 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:28:55.930 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:28:55.931 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:28:55.931 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:28:55.931 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:28:55.931 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:28:56.238 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:28:56.238 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:56.238 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:56.238 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:56.238 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[100622]: from='client.68687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[100622]: from='client.68693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3957738166' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3162540960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[102844]: from='client.68687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[102844]: from='client.68693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3957738166' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3162540960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:56 smithi196 ceph-mon[76349]: from='client.68687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:56 smithi196 ceph-mon[76349]: from='client.68693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3957738166' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3162540960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[102844]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[102844]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[102844]: from='client.68711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[100622]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[100622]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[100622]: from='client.68711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:57 smithi196 ceph-mon[76349]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:57 smithi196 ceph-mon[76349]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:57 smithi196 ceph-mon[76349]: from='client.68711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:59 smithi184 ceph-mon[100622]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:59 smithi184 ceph-mon[102844]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:59 smithi196 ceph-mon[76349]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:01 smithi184 ceph-mon[100622]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:01 smithi184 ceph-mon[102844]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:01.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:01.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:01.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:01 smithi196 ceph-mon[76349]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:03 smithi184 ceph-mon[100622]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:03 smithi184 ceph-mon[102844]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:03 smithi196 ceph-mon[76349]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:05 smithi184 ceph-mon[100622]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:05 smithi184 ceph-mon[102844]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:05 smithi196 ceph-mon[76349]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:06 smithi196 ceph-mon[76349]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[100622]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[102844]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:08.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:08 smithi184 ceph-mon[100622]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:08 smithi184 ceph-mon[102844]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:08 smithi196 ceph-mon[76349]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:10 smithi184 ceph-mon[100622]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:10 smithi184 ceph-mon[102844]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:11.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:11.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:11.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:10 smithi196 ceph-mon[76349]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:29:13.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:12 smithi196 ceph-mon[76349]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:12 smithi184 ceph-mon[100622]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:12 smithi184 ceph-mon[102844]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:14.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:15.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:14 smithi196 ceph-mon[76349]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:14 smithi184 ceph-mon[100622]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:14 smithi184 ceph-mon[102844]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:16 smithi184 ceph-mon[100622]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:16 smithi184 ceph-mon[102844]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:16 smithi196 ceph-mon[76349]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:18 smithi184 ceph-mon[100622]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:18 smithi184 ceph-mon[102844]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:18 smithi196 ceph-mon[76349]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:20 smithi184 ceph-mon[100622]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:20 smithi184 ceph-mon[102844]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:21.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:21.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:21.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:20 smithi196 ceph-mon[76349]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:29:23.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:22 smithi184 ceph-mon[100622]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:23.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:22 smithi184 ceph-mon[102844]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:22 smithi196 ceph-mon[76349]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:25.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:24 smithi184 ceph-mon[100622]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:25.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:24 smithi184 ceph-mon[102844]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:24 smithi196 ceph-mon[76349]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:26.535 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:29:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:26 smithi196 ceph-mon[76349]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[102844]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[100622]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 3h - - 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 20s ago 3h - - 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 1s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 20s ago 3h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 1s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 20s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:29:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 1s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 1s ago 3h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 20s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 1s ago 3h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 1s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 20s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 1s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 1s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 1s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 20s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 20s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 20s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:29:27.058 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 20s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:29:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 20s ago 3h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:29:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:29:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 20s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:29:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:29:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 20s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:29:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:29:27.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:29:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:29:27.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:29:27.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:29:27.643 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:29:27.643 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:29:27.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='client.58761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='client.68729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='client.58773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2550914674' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1574178927' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='client.58761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='client.68729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='client.58773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2550914674' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:28.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1574178927' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='client.58761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='client.68729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='client.58773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:28.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2550914674' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:28.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1574178927' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:29 smithi196 ceph-mon[76349]: from='client.68747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:29 smithi196 ceph-mon[76349]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[102844]: from='client.68747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[102844]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[100622]: from='client.68747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[100622]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:31 smithi196 ceph-mon[76349]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:31 smithi184 ceph-mon[100622]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:31 smithi184 ceph-mon[102844]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:31.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:31.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:31.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:29:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:33 smithi196 ceph-mon[76349]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:33.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:33 smithi184 ceph-mon[100622]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:33.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:33 smithi184 ceph-mon[102844]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:35.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:35 smithi196 ceph-mon[76349]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:35 smithi184 ceph-mon[100622]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:35 smithi184 ceph-mon[102844]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:37 smithi196 ceph-mon[76349]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:37 smithi184 ceph-mon[100622]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:37 smithi184 ceph-mon[102844]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:39 smithi196 ceph-mon[76349]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:39 smithi184 ceph-mon[100622]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:39 smithi184 ceph-mon[102844]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:41 smithi196 ceph-mon[76349]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:41 smithi184 ceph-mon[100622]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:41 smithi184 ceph-mon[102844]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:41.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:41.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:41.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:29:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:43 smithi196 ceph-mon[76349]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:29:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:43 smithi184 ceph-mon[100622]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:29:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:43 smithi184 ceph-mon[102844]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:29:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:45 smithi196 ceph-mon[76349]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:45 smithi184 ceph-mon[100622]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:45 smithi184 ceph-mon[102844]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:46 smithi196 ceph-mon[76349]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[100622]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:46 smithi184 ceph-mon[102844]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:29:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:49 smithi184 ceph-mon[100622]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:49 smithi184 ceph-mon[102844]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:49 smithi196 ceph-mon[76349]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:51.462 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:51 smithi196 ceph-mon[76349]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:51 smithi184 ceph-mon[100622]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:51 smithi184 ceph-mon[102844]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:51.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:51.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:29:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:29:51.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:29:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:29:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:29:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:53 smithi184 ceph-mon[100622]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:53 smithi184 ceph-mon[102844]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:53 smithi196 ceph-mon[76349]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:55 smithi184 ceph-mon[100622]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:55 smithi184 ceph-mon[102844]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:55 smithi196 ceph-mon[76349]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:57 smithi184 ceph-mon[100622]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:57 smithi184 ceph-mon[102844]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:57 smithi196 ceph-mon[76349]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:58.252 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:29:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:29:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T23:29:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 3h - - 2024-09-20T23:29:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:29:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 3h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 3h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 13s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 13s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:29:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 67.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:29:58.778 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:29:59.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:29:59.086 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:29:59.358 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:29:59.358 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:29:59.358 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:29:59.358 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:29:59.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:29:59.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[102844]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[102844]: from='client.68759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3049335849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[100622]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[100622]: from='client.68759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:29:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3049335849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:59 smithi196 ceph-mon[76349]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:59 smithi196 ceph-mon[76349]: from='client.68759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:29:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3049335849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:59.675 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:29:59.675 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:59.675 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:59.675 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:59.675 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[102844]: from='client.58803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[102844]: from='client.68771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3273547923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[100622]: from='client.58803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[100622]: from='client.68771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3273547923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:00 smithi196 ceph-mon[76349]: from='client.58803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:00 smithi196 ceph-mon[76349]: from='client.68771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3273547923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-mon[100622]: from='client.68783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-mon[100622]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-mon[102844]: from='client.68783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-mon[102844]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:01.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:01.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:01.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:01.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:01 smithi196 ceph-mon[76349]: from='client.68783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:01 smithi196 ceph-mon[76349]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T23:30:03.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:03 smithi184 ceph-mon[100622]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:03 smithi184 ceph-mon[102844]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:03 smithi196 ceph-mon[76349]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:05.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:05 smithi184 ceph-mon[100622]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:05.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:05 smithi184 ceph-mon[102844]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:05 smithi196 ceph-mon[76349]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[100622]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:06 smithi184 ceph-mon[102844]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:06 smithi196 ceph-mon[76349]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:09 smithi196 ceph-mon[76349]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:09 smithi184 ceph-mon[100622]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:09 smithi184 ceph-mon[102844]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:11.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:11.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:11.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:11.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:11.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:11.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:11 smithi196 ceph-mon[76349]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:11 smithi184 ceph-mon[100622]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:11 smithi184 ceph-mon[102844]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-20T23:30:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:13 smithi196 ceph-mon[76349]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:13 smithi184 ceph-mon[100622]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:13 smithi184 ceph-mon[102844]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:15 smithi196 ceph-mon[76349]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:15 smithi184 ceph-mon[100622]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:15 smithi184 ceph-mon[102844]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:16.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:16 smithi196 ceph-mon[76349]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:16 smithi184 ceph-mon[100622]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:16 smithi184 ceph-mon[102844]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:18 smithi196 ceph-mon[76349]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:18 smithi184 ceph-mon[100622]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:18 smithi184 ceph-mon[102844]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:20 smithi196 ceph-mon[76349]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:20 smithi184 ceph-mon[100622]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:20 smithi184 ceph-mon[102844]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:21.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:21.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:21.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:30:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:22 smithi196 ceph-mon[76349]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:22 smithi184 ceph-mon[100622]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:22 smithi184 ceph-mon[102844]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:24 smithi196 ceph-mon[76349]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:25.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:24 smithi184 ceph-mon[100622]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:25.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:24 smithi184 ceph-mon[102844]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[100622]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[102844]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:26 smithi196 ceph-mon[76349]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:28 smithi184 ceph-mon[100622]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:28 smithi184 ceph-mon[102844]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:28 smithi196 ceph-mon[76349]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:29.974 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 3h - - 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 4s ago 3h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 4s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:30:30.497 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 4s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 4s ago 3h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 3s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 4s ago 3h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 4s ago 3h 18.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 4s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 4s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 3s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 3s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:30:30.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 3s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:30:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 3s ago 3h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:30:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:30:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:30:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:30:30.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 3s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:30:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:30:30.810 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[102844]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[102844]: from='client.58833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[102844]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2047602189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[100622]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[100622]: from='client.58833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[100622]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.071 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2047602189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:30:31.079 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:30:31.080 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:30:31.386 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:31.386 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:31.387 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:31.387 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:31.387 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:31.387 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:31.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:31.387 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:31.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:31.387 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:31.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:30 smithi196 ceph-mon[76349]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:30 smithi196 ceph-mon[76349]: from='client.58833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:30 smithi196 ceph-mon[76349]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2047602189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:31 smithi196 ceph-mon[76349]: from='client.68807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:31 smithi196 ceph-mon[76349]: from='client.68819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3598700087' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[100622]: from='client.68807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[100622]: from='client.68819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3598700087' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:30:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[102844]: from='client.68807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[102844]: from='client.68819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3598700087' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:32 smithi184 ceph-mon[100622]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:32 smithi184 ceph-mon[102844]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:32 smithi196 ceph-mon[76349]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:34 smithi184 ceph-mon[100622]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:34 smithi184 ceph-mon[102844]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:34 smithi196 ceph-mon[76349]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:36 smithi196 ceph-mon[76349]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:36 smithi184 ceph-mon[102844]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:36 smithi184 ceph-mon[100622]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:38.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:38 smithi196 ceph-mon[76349]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:38 smithi184 ceph-mon[100622]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:38 smithi184 ceph-mon[102844]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:40 smithi184 ceph-mon[100622]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:40 smithi184 ceph-mon[102844]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:41.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:41.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:41.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:40 smithi196 ceph-mon[76349]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:30:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:42 smithi184 ceph-mon[100622]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:42 smithi184 ceph-mon[102844]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:42 smithi196 ceph-mon[76349]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:44 smithi184 ceph-mon[100622]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:44 smithi184 ceph-mon[102844]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:44 smithi196 ceph-mon[76349]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:47.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[100622]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[102844]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:47 smithi196 ceph-mon[76349]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:30:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:30:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:49 smithi184 ceph-mon[100622]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:49 smithi184 ceph-mon[102844]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:49.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:49 smithi196 ceph-mon[76349]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:51.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:51 smithi196 ceph-mon[76349]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:51 smithi184 ceph-mon[100622]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:51 smithi184 ceph-mon[102844]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:51.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:51.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:30:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:30:51.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:30:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:30:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:30:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:53 smithi184 ceph-mon[102844]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:53 smithi184 ceph-mon[100622]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:53 smithi196 ceph-mon[76349]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:55 smithi184 ceph-mon[102844]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:55.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:55 smithi184 ceph-mon[100622]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:55 smithi196 ceph-mon[76349]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:57 smithi184 ceph-mon[102844]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:57 smithi184 ceph-mon[100622]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:57 smithi196 ceph-mon[76349]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:59 smithi184 ceph-mon[100622]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:59 smithi184 ceph-mon[102844]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:59 smithi196 ceph-mon[76349]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:01 smithi196 ceph-mon[76349]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:01 smithi184 ceph-mon[100622]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:01 smithi184 ceph-mon[102844]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:01.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:01.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:01.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:01.684 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 3h - - 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 3h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 3h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 3h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:31:02.211 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:31:02.212 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:31:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:31:02.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:31:02.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:31:02.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:31:02.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:31:02.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:31:02.792 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:31:02.792 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:31:02.792 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:31:02.792 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:31:02.792 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:31:02.793 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:31:03.107 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:31:03.107 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:03.107 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:03 smithi196 ceph-mon[76349]: from='client.58869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:03 smithi196 ceph-mon[76349]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:03 smithi196 ceph-mon[76349]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:03 smithi196 ceph-mon[76349]: from='client.68843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4257752537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[100622]: from='client.58869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[100622]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[100622]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[100622]: from='client.68843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4257752537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[102844]: from='client.58869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[102844]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[102844]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[102844]: from='client.68843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:03.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4257752537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:04 smithi196 ceph-mon[76349]: from='client.68855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3880138515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:04 smithi184 ceph-mon[100622]: from='client.68855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3880138515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:04 smithi184 ceph-mon[102844]: from='client.68855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3880138515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:05 smithi196 ceph-mon[76349]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:05 smithi184 ceph-mon[100622]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:05 smithi184 ceph-mon[102844]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[100622]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.482 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[102844]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:07 smithi196 ceph-mon[76349]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:08.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.329 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.330 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:08.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.330 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:09.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:09 smithi184 ceph-mon[100622]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:09 smithi184 ceph-mon[102844]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:09 smithi196 ceph-mon[76349]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:11 smithi184 ceph-mon[100622]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:11 smithi184 ceph-mon[102844]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:11.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:11.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:11.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:11 smithi196 ceph-mon[76349]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:31:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:13 smithi184 ceph-mon[100622]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:13 smithi184 ceph-mon[102844]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:13 smithi196 ceph-mon[76349]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:15 smithi184 ceph-mon[100622]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:15 smithi184 ceph-mon[102844]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:15 smithi196 ceph-mon[76349]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:17 smithi196 ceph-mon[76349]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:17 smithi184 ceph-mon[100622]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:17 smithi184 ceph-mon[102844]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:18.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:19 smithi196 ceph-mon[76349]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:19 smithi184 ceph-mon[100622]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:19 smithi184 ceph-mon[102844]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:21 smithi196 ceph-mon[76349]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:21 smithi184 ceph-mon[100622]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:21 smithi184 ceph-mon[102844]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:21.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:21.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:21.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:31:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:23 smithi184 ceph-mon[100622]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:23 smithi184 ceph-mon[102844]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:23 smithi196 ceph-mon[76349]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:25 smithi184 ceph-mon[100622]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:25 smithi184 ceph-mon[102844]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:25 smithi196 ceph-mon[76349]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[100622]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[102844]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:27 smithi196 ceph-mon[76349]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.387 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:28.387 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.388 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:29.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:29 smithi184 ceph-mon[100622]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:29.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:29.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:29 smithi184 ceph-mon[102844]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:29.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:29 smithi196 ceph-mon[76349]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:31 smithi196 ceph-mon[76349]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:31 smithi184 ceph-mon[100622]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:31 smithi184 ceph-mon[102844]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:31.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:31.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:31.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:31:33.403 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:31:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:33 smithi184 ceph-mon[100622]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:33 smithi184 ceph-mon[102844]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:33 smithi196 ceph-mon[76349]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 3h - - 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 6s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:31:33.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 3h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 6s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 6s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 6s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 6s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 6s ago 3h 68.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:31:33.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:31:33.938 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 6s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:34.249 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:31:34.250 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:31:34.519 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:31:34.520 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:31:34.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1037398904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:34.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1037398904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1037398904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:34.829 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:31:34.830 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:34.830 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:34.830 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[100622]: from='client.68867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[100622]: from='client.68873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[100622]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[100622]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3649468294' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[102844]: from='client.68867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[102844]: from='client.68873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[102844]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[102844]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3649468294' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:35 smithi196 ceph-mon[76349]: from='client.68867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:35 smithi196 ceph-mon[76349]: from='client.68873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:35 smithi196 ceph-mon[76349]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:35 smithi196 ceph-mon[76349]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3649468294' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:36 smithi196 ceph-mon[76349]: from='client.58929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:36 smithi184 ceph-mon[100622]: from='client.58929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:36 smithi184 ceph-mon[102844]: from='client.58929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:37.635 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:37 smithi196 ceph-mon[76349]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:37 smithi184 ceph-mon[100622]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:37 smithi184 ceph-mon[102844]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:39.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:39 smithi196 ceph-mon[76349]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:39 smithi184 ceph-mon[102844]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:39 smithi184 ceph-mon[100622]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:41 smithi184 ceph-mon[100622]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:41 smithi184 ceph-mon[102844]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:41.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:41.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:41.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:41 smithi196 ceph-mon[76349]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:31:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:43 smithi196 ceph-mon[76349]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:43 smithi184 ceph-mon[100622]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:43 smithi184 ceph-mon[102844]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:45 smithi196 ceph-mon[76349]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:45 smithi184 ceph-mon[100622]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:45 smithi184 ceph-mon[102844]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:47.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[102844]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:47.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[100622]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:47.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:47 smithi196 ceph-mon[76349]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:48.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:48.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:31:49.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:49 smithi184 ceph-mon[100622]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:49.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:49 smithi184 ceph-mon[102844]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:49 smithi196 ceph-mon[76349]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:51.340 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:51.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:51.340 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:51.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:31:51.340 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:31:51.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:31:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:51 smithi196 ceph-mon[76349]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:51 smithi184 ceph-mon[100622]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:51 smithi184 ceph-mon[102844]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:31:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:31:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:53 smithi196 ceph-mon[76349]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:53 smithi184 ceph-mon[100622]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:53 smithi184 ceph-mon[102844]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:55 smithi196 ceph-mon[76349]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:55 smithi184 ceph-mon[100622]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:55 smithi184 ceph-mon[102844]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:57 smithi196 ceph-mon[76349]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:57 smithi184 ceph-mon[100622]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:57 smithi184 ceph-mon[102844]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:59 smithi196 ceph-mon[76349]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:59 smithi184 ceph-mon[100622]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:59 smithi184 ceph-mon[102844]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:01.358 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:01.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:01.359 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:01.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:01.359 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:01.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:01 smithi196 ceph-mon[76349]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:01 smithi184 ceph-mon[100622]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:01 smithi184 ceph-mon[102844]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:32:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:03 smithi196 ceph-mon[76349]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:03 smithi184 ceph-mon[100622]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:03 smithi184 ceph-mon[102844]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:05.126 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:32:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:05 smithi184 ceph-mon[100622]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:05 smithi184 ceph-mon[102844]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:05 smithi196 ceph-mon[76349]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:05.655 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:32:05.655 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 3h - - 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 3h - - 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 19s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 3h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 3h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 19s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:32:05.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 69.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:32:05.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:32:05.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:32:05.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:32:05.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:32:05.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:32:05.969 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:32:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:32:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:32:06.238 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:32:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:06 smithi184 ceph-mon[102844]: from='client.58941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1605581064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:06 smithi184 ceph-mon[100622]: from='client.58941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1605581064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:06.575 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:32:06.575 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:06.575 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:06 smithi196 ceph-mon[76349]: from='client.58941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1605581064' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='client.68909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='client.68915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='client.58965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1885761610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='client.68909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='client.68915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='client.58965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1885761610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='client.68909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='client.68915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='client.58965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:07.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1885761610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[100622]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[102844]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:09 smithi196 ceph-mon[76349]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:11 smithi196 ceph-mon[76349]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:11 smithi184 ceph-mon[100622]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:11 smithi184 ceph-mon[102844]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:11.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:11.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:11.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:32:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:13 smithi184 ceph-mon[102844]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:13 smithi184 ceph-mon[100622]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:13 smithi196 ceph-mon[76349]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:15 smithi196 ceph-mon[76349]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:15 smithi184 ceph-mon[100622]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:15 smithi184 ceph-mon[102844]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:17 smithi196 ceph-mon[76349]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:17 smithi184 ceph-mon[100622]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:17 smithi184 ceph-mon[102844]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:19.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:19 smithi196 ceph-mon[76349]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:19 smithi184 ceph-mon[100622]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:19 smithi184 ceph-mon[102844]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:21 smithi196 ceph-mon[76349]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:21 smithi184 ceph-mon[100622]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:21 smithi184 ceph-mon[102844]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:21.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:21.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:21.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:32:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:23 smithi196 ceph-mon[76349]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:23 smithi184 ceph-mon[100622]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:23 smithi184 ceph-mon[102844]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:25 smithi196 ceph-mon[76349]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:25 smithi184 ceph-mon[100622]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:25 smithi184 ceph-mon[102844]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:27 smithi196 ceph-mon[76349]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[102844]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[100622]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:29.497 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[100622]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[102844]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.498 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:29 smithi196 ceph-mon[76349]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:31.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:31 smithi184 ceph-mon[100622]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:31.515 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:31.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:31.515 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:31.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:31.515 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:31.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:31.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:31 smithi184 ceph-mon[102844]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:31 smithi196 ceph-mon[76349]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:32:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:33 smithi184 ceph-mon[100622]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:33 smithi184 ceph-mon[102844]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:33 smithi196 ceph-mon[76349]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:35 smithi184 ceph-mon[102844]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:35 smithi184 ceph-mon[100622]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:35 smithi196 ceph-mon[76349]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:36.870 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 3h - - 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:32:37.398 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 3h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 3h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:32:37.399 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 3h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:32:37.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:32:37.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:32:37.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:32:37.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:32:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:37 smithi184 ceph-mon[100622]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:37 smithi184 ceph-mon[102844]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:37 smithi196 ceph-mon[76349]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:37.707 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:32:37.708 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:32:37.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:32:37.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:32:37.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:32:37.977 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:32:37.978 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:32:38.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:38.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[100622]: from='client.68939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[100622]: from='client.68945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3589835400' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:38.291 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:32:38.291 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:38.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:38.291 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:38.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[102844]: from='client.68939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[102844]: from='client.68945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3589835400' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:38 smithi196 ceph-mon[76349]: from='client.68939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:38 smithi196 ceph-mon[76349]: from='client.68945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3589835400' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[100622]: from='client.68951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[100622]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[100622]: from='client.59001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4271983647' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[102844]: from='client.68951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[102844]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[102844]: from='client.59001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4271983647' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:39 smithi196 ceph-mon[76349]: from='client.68951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:39 smithi196 ceph-mon[76349]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:39 smithi196 ceph-mon[76349]: from='client.59001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4271983647' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:41.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:41 smithi196 ceph-mon[76349]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:41 smithi184 ceph-mon[100622]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:41 smithi184 ceph-mon[102844]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:41.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:41.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:41.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:32:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:43 smithi184 ceph-mon[100622]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:43 smithi184 ceph-mon[102844]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:43 smithi196 ceph-mon[76349]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:45 smithi184 ceph-mon[100622]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:45 smithi184 ceph-mon[102844]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:45 smithi196 ceph-mon[76349]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[100622]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[102844]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:47 smithi196 ceph-mon[76349]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:49.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[100622]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:49.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[102844]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:49.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:49 smithi196 ceph-mon[76349]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:32:51.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:51.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:51.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:51.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:32:51.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:32:51.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:32:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:51 smithi184 ceph-mon[100622]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:51 smithi184 ceph-mon[102844]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:51 smithi196 ceph-mon[76349]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:52.152 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:32:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:32:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:53 smithi196 ceph-mon[76349]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:53 smithi184 ceph-mon[100622]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:53 smithi184 ceph-mon[102844]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:55 smithi196 ceph-mon[76349]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:55 smithi184 ceph-mon[100622]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:55 smithi184 ceph-mon[102844]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:57 smithi196 ceph-mon[76349]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:57 smithi184 ceph-mon[100622]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:57 smithi184 ceph-mon[102844]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:59 smithi196 ceph-mon[76349]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:59 smithi184 ceph-mon[100622]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:59 smithi184 ceph-mon[102844]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:01.358 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:01.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:01.358 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:01.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:01.358 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:01.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:01.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:01 smithi196 ceph-mon[76349]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:01 smithi184 ceph-mon[100622]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:01 smithi184 ceph-mon[102844]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:33:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:03 smithi196 ceph-mon[76349]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:03.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:03 smithi184 ceph-mon[100622]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:03.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:03 smithi184 ceph-mon[102844]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:05 smithi196 ceph-mon[76349]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:05 smithi184 ceph-mon[100622]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:05 smithi184 ceph-mon[102844]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:07 smithi196 ceph-mon[76349]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[102844]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[100622]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:08.594 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:33:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 3h - - 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 2s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 3h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:33:09.122 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 0s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 3h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 0s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 2s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 0s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:33:09.123 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 0s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 0s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 0s ago 3h 70.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 2s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:33:09.124 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:33:09.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[100622]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:09.372 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:09.432 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:33:09.433 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:33:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:09 smithi196 ceph-mon[76349]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:33:09.699 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:33:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:33:09.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[102844]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:09.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:09.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:10.012 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:33:10.012 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:10.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:10.012 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:10.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[100622]: from='client.59013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[100622]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[100622]: from='client.59025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2820173807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3215462468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[102844]: from='client.59013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[102844]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[102844]: from='client.59025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2820173807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:10.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3215462468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:10 smithi196 ceph-mon[76349]: from='client.59013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:10 smithi196 ceph-mon[76349]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:10 smithi196 ceph-mon[76349]: from='client.59025 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2820173807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3215462468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:11.379 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:11.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:11.379 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:11.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:11.379 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:11.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:11 smithi196 ceph-mon[76349]: from='client.59037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:11 smithi196 ceph-mon[76349]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:11.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-mon[100622]: from='client.59037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:11.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-mon[100622]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:11.653 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-mon[102844]: from='client.59037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:11.653 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:11 smithi184 ceph-mon[102844]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:33:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:12 smithi196 ceph-mon[76349]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:12.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:12 smithi184 ceph-mon[100622]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:12.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:12 smithi184 ceph-mon[102844]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:14 smithi184 ceph-mon[100622]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:14 smithi184 ceph-mon[102844]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:14 smithi196 ceph-mon[76349]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:16 smithi184 ceph-mon[100622]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:16 smithi184 ceph-mon[102844]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:16 smithi196 ceph-mon[76349]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:18 smithi184 ceph-mon[100622]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:18 smithi184 ceph-mon[102844]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:18 smithi196 ceph-mon[76349]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:21.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:20 smithi196 ceph-mon[76349]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:20 smithi184 ceph-mon[100622]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:21 smithi184 ceph-mon[102844]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:21.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:21.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:21.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:22.258 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:33:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:23 smithi184 ceph-mon[100622]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:23 smithi184 ceph-mon[102844]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:23 smithi196 ceph-mon[76349]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:25 smithi184 ceph-mon[100622]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:25 smithi184 ceph-mon[102844]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:25 smithi196 ceph-mon[76349]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:27 smithi184 ceph-mon[100622]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:33:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:27 smithi184 ceph-mon[102844]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:33:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:27 smithi196 ceph-mon[76349]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:33:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:29 smithi196 ceph-mon[76349]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[100622]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[102844]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:31.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:31 smithi184 ceph-mon[100622]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:31.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:31 smithi184 ceph-mon[102844]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:31.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:31.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:31.260 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:31.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:31.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:31.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:31 smithi196 ceph-mon[76349]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:32.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:33:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:33 smithi184 ceph-mon[100622]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:33 smithi184 ceph-mon[102844]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:33 smithi196 ceph-mon[76349]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:35 smithi196 ceph-mon[76349]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:35 smithi184 ceph-mon[100622]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:35 smithi184 ceph-mon[102844]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:37 smithi196 ceph-mon[76349]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:37 smithi184 ceph-mon[100622]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:37 smithi184 ceph-mon[102844]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:39 smithi196 ceph-mon[76349]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:39 smithi184 ceph-mon[100622]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:39 smithi184 ceph-mon[102844]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:40.310 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 3h - - 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 3h - - 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 3h 389M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:33:40.835 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 3h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 13s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 13s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:33:40.836 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:33:41.146 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:33:41.146 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:33:41.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:33:41.148 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:33:41.148 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:33:41.363 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:41 smithi196 ceph-mon[76349]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:41.363 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:41 smithi196 ceph-mon[76349]: from='client.59049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:41.416 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:33:41.417 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:33:41.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-mon[100622]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:41.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-mon[100622]: from='client.59049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:41.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-mon[102844]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:41.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-mon[102844]: from='client.59049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:41.418 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:41.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:41.418 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:41.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:41.418 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:41.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:41.727 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:33:41.727 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:41.727 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:41.727 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[100622]: from='client.59055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[100622]: from='client.69023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/98194662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2607736341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[102844]: from='client.59055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[102844]: from='client.69023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/98194662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:42.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2607736341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:42 smithi196 ceph-mon[76349]: from='client.59055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:42 smithi196 ceph-mon[76349]: from='client.69023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/98194662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2607736341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:43 smithi196 ceph-mon[76349]: from='client.59073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:43 smithi196 ceph-mon[76349]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:43 smithi184 ceph-mon[100622]: from='client.59073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:43 smithi184 ceph-mon[100622]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:43 smithi184 ceph-mon[102844]: from='client.59073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:43 smithi184 ceph-mon[102844]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:45 smithi196 ceph-mon[76349]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:45 smithi184 ceph-mon[100622]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:45 smithi184 ceph-mon[102844]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:47 smithi196 ceph-mon[76349]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:47 smithi184 ceph-mon[100622]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:47 smithi184 ceph-mon[102844]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[100622]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[102844]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:49 smithi196 ceph-mon[76349]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:33:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:51 smithi184 ceph-mon[100622]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:51 smithi184 ceph-mon[102844]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:51.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:51.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:33:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:33:51.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:33:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:51 smithi196 ceph-mon[76349]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:52.007 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:33:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:33:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:53 smithi184 ceph-mon[100622]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:53 smithi184 ceph-mon[102844]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:53 smithi196 ceph-mon[76349]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:55 smithi184 ceph-mon[100622]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:55 smithi184 ceph-mon[102844]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:55 smithi196 ceph-mon[76349]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:57 smithi184 ceph-mon[102844]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:57 smithi184 ceph-mon[100622]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:57 smithi196 ceph-mon[76349]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:59 smithi184 ceph-mon[100622]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:59 smithi184 ceph-mon[102844]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:59 smithi196 ceph-mon[76349]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:59.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:01 smithi184 ceph-mon[100622]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:01 smithi184 ceph-mon[102844]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:01.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:01.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:01.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:01 smithi196 ceph-mon[76349]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:34:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:03 smithi184 ceph-mon[100622]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:03 smithi184 ceph-mon[102844]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:03 smithi196 ceph-mon[76349]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:05 smithi196 ceph-mon[76349]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:05 smithi184 ceph-mon[100622]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:05 smithi184 ceph-mon[102844]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:07 smithi184 ceph-mon[102844]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:34:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:07 smithi184 ceph-mon[100622]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:34:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:07 smithi196 ceph-mon[76349]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:34:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:09 smithi196 ceph-mon[76349]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[100622]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:34:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[102844]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:34:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:11.349 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:11.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:11.349 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:11.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:11.349 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:11.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:11 smithi196 ceph-mon[76349]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:34:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:11 smithi184 ceph-mon[100622]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:34:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:11 smithi184 ceph-mon[102844]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:34:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:34:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 3h - - 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 3s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 3h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:34:12.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 3s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 3h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 3h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 3h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 3s ago 3h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 3s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 3s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 3s ago 3h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:34:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 3s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:34:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:34:12.893 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:34:13.169 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:34:13.169 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:34:13.169 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:34:13.169 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:34:13.169 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:34:13.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:34:13.494 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:34:13.494 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:13.494 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:13.494 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:13.494 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:13 smithi196 ceph-mon[76349]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:34:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:13 smithi196 ceph-mon[76349]: from='client.69047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:13 smithi196 ceph-mon[76349]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2835935221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:13.661 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[100622]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[100622]: from='client.69047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[100622]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2835935221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[102844]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[102844]: from='client.69047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[102844]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:13.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2835935221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:14 smithi196 ceph-mon[76349]: from='client.69059 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:14 smithi196 ceph-mon[76349]: from='client.59109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1817941165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[100622]: from='client.69059 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[100622]: from='client.59109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1817941165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[102844]: from='client.69059 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[102844]: from='client.59109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:14.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1817941165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:14.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:15 smithi196 ceph-mon[76349]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:34:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:15 smithi184 ceph-mon[102844]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:34:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:15 smithi184 ceph-mon[100622]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:34:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:17 smithi196 ceph-mon[76349]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:34:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:17 smithi184 ceph-mon[100622]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:17 smithi184 ceph-mon[102844]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:34:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:19 smithi196 ceph-mon[76349]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-20T23:34:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:19 smithi184 ceph-mon[102844]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-20T23:34:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:19 smithi184 ceph-mon[100622]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-20T23:34:21.375 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:21.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:21.375 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:21.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:21.375 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:21.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:21 smithi196 ceph-mon[76349]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T23:34:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:21 smithi184 ceph-mon[102844]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T23:34:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:21 smithi184 ceph-mon[100622]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-20T23:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:34:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:23 smithi196 ceph-mon[76349]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T23:34:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:23 smithi184 ceph-mon[100622]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T23:34:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:23 smithi184 ceph-mon[102844]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T23:34:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:25 smithi196 ceph-mon[76349]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:34:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:25 smithi184 ceph-mon[100622]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:34:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:25 smithi184 ceph-mon[102844]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T23:34:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:27 smithi196 ceph-mon[76349]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:34:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:27 smithi184 ceph-mon[100622]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:34:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:27 smithi184 ceph-mon[102844]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:34:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:29 smithi196 ceph-mon[76349]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:34:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[102844]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[100622]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:34:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:30 smithi184 ceph-mon[100622]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T23:34:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:30 smithi184 ceph-mon[102844]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T23:34:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:30 smithi196 ceph-mon[76349]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-20T23:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:31.089Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:31.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:31.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:34:33.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:32 smithi184 ceph-mon[100622]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:34:33.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:32 smithi184 ceph-mon[102844]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:34:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:32 smithi196 ceph-mon[76349]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:34:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:34 smithi184 ceph-mon[100622]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:34:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:34 smithi184 ceph-mon[102844]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:34:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:34 smithi196 ceph-mon[76349]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:34:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:36 smithi184 ceph-mon[100622]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:34:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:36 smithi184 ceph-mon[102844]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:34:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:36 smithi196 ceph-mon[76349]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-20T23:34:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:38 smithi184 ceph-mon[100622]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T23:34:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:38 smithi184 ceph-mon[102844]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T23:34:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:38 smithi196 ceph-mon[76349]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T23:34:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:40 smithi184 ceph-mon[100622]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:34:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:40 smithi184 ceph-mon[102844]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:34:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:41.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:41.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:41.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:40 smithi196 ceph-mon[76349]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:34:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:34:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:42 smithi184 ceph-mon[100622]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:34:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:42 smithi184 ceph-mon[102844]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:34:43.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:42 smithi196 ceph-mon[76349]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:34:43.792 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:34:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:44.336 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:34:44.336 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 3h - - 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 3h - - 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 15s ago 3h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 15s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 3h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 15s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 3h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 15s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 3h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:34:44.337 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 72.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 90.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 15s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 15s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 15s ago 3h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 15s ago 3h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 15s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:34:44.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 15s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:34:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:34:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:34:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:34:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:34:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:34:44.648 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:34:44.914 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:34:44.915 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:34:44.916 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:34:45.227 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:34:45.227 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:45.227 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:45.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[100622]: from='client.59121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[100622]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:34:45.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[100622]: from='client.59127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[100622]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1312619914' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[102844]: from='client.59121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[102844]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[102844]: from='client.59127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[102844]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1312619914' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:44 smithi196 ceph-mon[76349]: from='client.59121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:44 smithi196 ceph-mon[76349]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T23:34:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:44 smithi196 ceph-mon[76349]: from='client.59127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:44 smithi196 ceph-mon[76349]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1312619914' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:45 smithi184 ceph-mon[102844]: from='client.69107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3630411819' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:45 smithi184 ceph-mon[100622]: from='client.69107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3630411819' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:45 smithi196 ceph-mon[76349]: from='client.69107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3630411819' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:46 smithi184 ceph-mon[100622]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:34:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:46 smithi184 ceph-mon[102844]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:34:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:46 smithi196 ceph-mon[76349]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T23:34:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:48 smithi184 ceph-mon[100622]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:48 smithi184 ceph-mon[102844]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:48 smithi196 ceph-mon[76349]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:34:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:51 smithi184 ceph-mon[100622]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:51 smithi184 ceph-mon[102844]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:51.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:51.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:34:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:34:51.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:51 smithi196 ceph-mon[76349]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:34:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:34:53.487 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:53 smithi184 ceph-mon[100622]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:53.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:53 smithi184 ceph-mon[102844]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:53 smithi196 ceph-mon[76349]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:55 smithi184 ceph-mon[100622]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:55 smithi184 ceph-mon[102844]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:55 smithi196 ceph-mon[76349]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:57 smithi184 ceph-mon[100622]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:57 smithi184 ceph-mon[102844]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:57 smithi196 ceph-mon[76349]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:59 smithi184 ceph-mon[100622]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:59 smithi184 ceph-mon[102844]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:59 smithi196 ceph-mon[76349]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:01 smithi184 ceph-mon[100622]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:01 smithi184 ceph-mon[102844]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:01.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:01.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:01.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:01 smithi196 ceph-mon[76349]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:02] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:35:03.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:03 smithi184 ceph-mon[100622]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:03.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:03 smithi184 ceph-mon[102844]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:03 smithi196 ceph-mon[76349]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:05.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:05 smithi196 ceph-mon[76349]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:05 smithi184 ceph-mon[100622]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:05 smithi184 ceph-mon[102844]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:07 smithi184 ceph-mon[100622]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:07 smithi184 ceph-mon[102844]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:07 smithi196 ceph-mon[76349]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:09 smithi184 ceph-mon[100622]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:09 smithi184 ceph-mon[102844]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:09 smithi196 ceph-mon[76349]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:11 smithi184 ceph-mon[100622]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:11 smithi184 ceph-mon[102844]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:11.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:11.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:11.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:11 smithi196 ceph-mon[76349]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:12] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:35:13.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:13 smithi184 ceph-mon[100622]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:13.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:13 smithi184 ceph-mon[102844]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:13 smithi196 ceph-mon[76349]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:14.339 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:14.339 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:15.397 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:15 smithi184 ceph-mon[100622]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:15.398 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:15 smithi184 ceph-mon[102844]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:15.523 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:35:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:15 smithi196 ceph-mon[76349]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 3h - - 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 3h - - 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 6s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 6s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 3h 391M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 6s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 3h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:35:16.129 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 6s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 6s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 6s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 3h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 6s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 8s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:35:16.130 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 6s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:35:16.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:35:16.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:35:16.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:35:16.721 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:35:17.036 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:35:17.036 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:17.036 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: from='client.69119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: from='client.69125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: from='client.59169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4183552706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1956189061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: from='client.69119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: from='client.69125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: from='client.59169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4183552706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1956189061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: from='client.69119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: from='client.69125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: from='client.59169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4183552706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1956189061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:18 smithi184 ceph-mon[100622]: from='client.69143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:18 smithi184 ceph-mon[102844]: from='client.69143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:18 smithi196 ceph-mon[76349]: from='client.69143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:19 smithi196 ceph-mon[76349]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:19 smithi184 ceph-mon[100622]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:19 smithi184 ceph-mon[102844]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:21 smithi184 ceph-mon[100622]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:21 smithi184 ceph-mon[102844]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:21.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:21.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:21.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:21 smithi196 ceph-mon[76349]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:35:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:23 smithi196 ceph-mon[76349]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:23 smithi184 ceph-mon[100622]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:23 smithi184 ceph-mon[102844]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:25 smithi196 ceph-mon[76349]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:25 smithi184 ceph-mon[100622]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:25 smithi184 ceph-mon[102844]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:27.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:27 smithi196 ceph-mon[76349]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:27.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:27 smithi184 ceph-mon[100622]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:27.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:27 smithi184 ceph-mon[102844]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:29 smithi196 ceph-mon[76349]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:29 smithi184 ceph-mon[100622]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:29 smithi184 ceph-mon[102844]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:31.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:31.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:31.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:31.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:31.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:31.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:31 smithi184 ceph-mon[100622]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:31 smithi184 ceph-mon[102844]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:31 smithi196 ceph-mon[76349]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:35:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:33 smithi184 ceph-mon[100622]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:33 smithi184 ceph-mon[102844]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:33 smithi196 ceph-mon[76349]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:34.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:34 smithi184 ceph-mon[102844]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:34.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:34 smithi184 ceph-mon[100622]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:34 smithi196 ceph-mon[76349]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:36 smithi184 ceph-mon[100622]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:36 smithi184 ceph-mon[102844]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:36 smithi196 ceph-mon[76349]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:38 smithi184 ceph-mon[100622]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:38 smithi184 ceph-mon[102844]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:38 smithi196 ceph-mon[76349]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:40 smithi184 ceph-mon[100622]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:40 smithi184 ceph-mon[102844]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:41.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:41.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:41.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:40 smithi196 ceph-mon[76349]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:35:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:42 smithi184 ceph-mon[100622]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:42 smithi184 ceph-mon[102844]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:42 smithi196 ceph-mon[76349]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:45.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:45 smithi196 ceph-mon[76349]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:45 smithi184 ceph-mon[100622]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:45 smithi184 ceph-mon[102844]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:47 smithi184 ceph-mon[100622]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:47 smithi184 ceph-mon[102844]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:47.326 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:35:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:47 smithi196 ceph-mon[76349]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:47.854 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 20s ago 3h - - 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 3h - - 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 20s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 18s ago 3h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 20s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 20s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 20s ago 3h 392M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 3h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 20s ago 3h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 20s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 18s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 20s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:35:47.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 20s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 20s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 20s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 3h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 20s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 20s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:35:47.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[100622]: from='client.59193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:48.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[102844]: from='client.59193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:48.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:35:48.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:35:48.177 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:35:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:48 smithi196 ceph-mon[76349]: from='client.59193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:35:48.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:35:48.449 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:35:48.758 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:35:48.758 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:48.758 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:48.758 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:48.758 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[102844]: from='client.69161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[102844]: from='client.69167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[102844]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/577436743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/904679275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[100622]: from='client.69161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[100622]: from='client.69167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[100622]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/577436743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/904679275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:49 smithi196 ceph-mon[76349]: from='client.69161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:49 smithi196 ceph-mon[76349]: from='client.69167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:49 smithi196 ceph-mon[76349]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/577436743' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/904679275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[100622]: from='client.59217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[102844]: from='client.59217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:50 smithi196 ceph-mon[76349]: from='client.59217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:35:51.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:51 smithi196 ceph-mon[76349]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:51 smithi184 ceph-mon[100622]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:51 smithi184 ceph-mon[102844]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:51.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:51.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:35:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:35:51.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:35:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:35:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:53 smithi184 ceph-mon[100622]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:53 smithi184 ceph-mon[102844]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:53 smithi196 ceph-mon[76349]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:55 smithi184 ceph-mon[100622]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:55 smithi184 ceph-mon[102844]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:55 smithi196 ceph-mon[76349]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:57 smithi184 ceph-mon[100622]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:57 smithi184 ceph-mon[102844]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:57 smithi196 ceph-mon[76349]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:59 smithi184 ceph-mon[102844]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:59 smithi184 ceph-mon[100622]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:59.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:59 smithi196 ceph-mon[76349]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:01 smithi184 ceph-mon[100622]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:01 smithi184 ceph-mon[102844]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:01.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:01.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:01.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:01 smithi196 ceph-mon[76349]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:36:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:03 smithi196 ceph-mon[76349]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:03 smithi184 ceph-mon[100622]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:03 smithi184 ceph-mon[102844]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:05 smithi196 ceph-mon[76349]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:05 smithi184 ceph-mon[102844]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:05 smithi184 ceph-mon[100622]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:07 smithi196 ceph-mon[76349]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:07 smithi184 ceph-mon[100622]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:07 smithi184 ceph-mon[102844]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:08.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:09 smithi196 ceph-mon[76349]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[100622]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[102844]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[100622]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:10 smithi184 ceph-mon[102844]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:10 smithi196 ceph-mon[76349]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:11.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:11.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:11.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:36:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:12 smithi184 ceph-mon[100622]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:12 smithi184 ceph-mon[102844]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:12 smithi196 ceph-mon[76349]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:15 smithi184 ceph-mon[100622]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:15 smithi184 ceph-mon[102844]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:15 smithi196 ceph-mon[76349]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:17 smithi184 ceph-mon[100622]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:17 smithi184 ceph-mon[102844]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:17 smithi196 ceph-mon[76349]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:19.045 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:36:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:19 smithi184 ceph-mon[100622]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:19 smithi184 ceph-mon[102844]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:19 smithi196 ceph-mon[76349]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 3h - - 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 3h - - 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 11s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 3h 391M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:36:19.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 3h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 11s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 3h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 3h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:36:19.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:36:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:36:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:19.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:36:19.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:36:20.156 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:36:20.157 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[102844]: from='client.59229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[102844]: from='client.69197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/163244648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[100622]: from='client.59229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[100622]: from='client.69197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/163244648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:20 smithi196 ceph-mon[76349]: from='client.59229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:20 smithi196 ceph-mon[76349]: from='client.69197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/163244648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:20.468 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:36:20.469 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:20.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:20.469 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:20.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:21.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:21 smithi196 ceph-mon[76349]: from='client.69203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:21 smithi196 ceph-mon[76349]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:21.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:21 smithi196 ceph-mon[76349]: from='client.69215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1443085712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[100622]: from='client.69203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[100622]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[100622]: from='client.69215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1443085712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[102844]: from='client.69203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[102844]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[102844]: from='client.69215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1443085712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:21.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:21.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:21.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:36:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:23 smithi184 ceph-mon[100622]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:23 smithi184 ceph-mon[102844]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:23 smithi196 ceph-mon[76349]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:25 smithi184 ceph-mon[100622]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:25 smithi184 ceph-mon[102844]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:25 smithi196 ceph-mon[76349]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:27 smithi184 ceph-mon[100622]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:27 smithi184 ceph-mon[102844]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:27 smithi196 ceph-mon[76349]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:28.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:29 smithi196 ceph-mon[76349]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[100622]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[102844]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[100622]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:30 smithi184 ceph-mon[102844]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:30 smithi196 ceph-mon[76349]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:31.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:31.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:31.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:36:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:33 smithi184 ceph-mon[100622]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:33 smithi184 ceph-mon[102844]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:33 smithi196 ceph-mon[76349]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:35 smithi184 ceph-mon[100622]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:35 smithi184 ceph-mon[102844]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:35 smithi196 ceph-mon[76349]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:37 smithi184 ceph-mon[100622]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:37 smithi184 ceph-mon[102844]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:37 smithi196 ceph-mon[76349]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:39 smithi184 ceph-mon[100622]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:39 smithi184 ceph-mon[102844]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:39 smithi196 ceph-mon[76349]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:41 smithi184 ceph-mon[100622]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:41 smithi184 ceph-mon[102844]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:41.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:41.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:41.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:41 smithi196 ceph-mon[76349]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:36:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:43 smithi184 ceph-mon[100622]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:43 smithi184 ceph-mon[102844]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:43 smithi196 ceph-mon[76349]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:45 smithi184 ceph-mon[100622]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:45 smithi184 ceph-mon[102844]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:45 smithi196 ceph-mon[76349]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:47 smithi184 ceph-mon[102844]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:47 smithi184 ceph-mon[100622]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:47.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:47 smithi196 ceph-mon[76349]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:48.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:49.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:49 smithi196 ceph-mon[76349]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:49.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[100622]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[102844]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:50.767 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:36:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[100622]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:50 smithi184 ceph-mon[102844]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:36:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:50 smithi196 ceph-mon[76349]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 3h - - 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 3h - - 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 2s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 1s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:36:51.292 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 3h 393M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 3h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 2s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 1s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:36:51.293 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:36:51.294 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 3h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:36:51.294 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:36:51.294 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:36:51.294 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 2s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:36:51.294 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:36:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:51.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:51.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:36:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:36:51.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:51.603 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:36:51.604 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:36:51.874 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:36:51.875 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:36:52.186 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:36:52.186 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:52.186 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:52.186 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:52.186 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:36:52] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[100622]: from='client.69227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[100622]: from='client.59271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[100622]: from='client.69239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2382764521' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[102844]: from='client.69227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[102844]: from='client.59271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[102844]: from='client.69239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2382764521' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:51 smithi196 ceph-mon[76349]: from='client.69227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:51 smithi196 ceph-mon[76349]: from='client.59271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:51 smithi196 ceph-mon[76349]: from='client.69239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2382764521' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[100622]: from='client.69251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[100622]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/303757487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[102844]: from='client.69251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[102844]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:53.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/303757487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:52 smithi196 ceph-mon[76349]: from='client.69251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:52 smithi196 ceph-mon[76349]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/303757487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:55 smithi184 ceph-mon[100622]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:55 smithi184 ceph-mon[102844]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:55 smithi196 ceph-mon[76349]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:57 smithi184 ceph-mon[100622]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:57 smithi184 ceph-mon[102844]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:57 smithi196 ceph-mon[76349]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:59.150 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:59 smithi184 ceph-mon[100622]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:59.150 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:59.151 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:59 smithi184 ceph-mon[102844]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:59.151 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:59 smithi196 ceph-mon[76349]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:01 smithi184 ceph-mon[100622]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:01 smithi184 ceph-mon[102844]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:01.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:01.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:01.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:01 smithi196 ceph-mon[76349]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:37:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:03 smithi184 ceph-mon[100622]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:03 smithi184 ceph-mon[102844]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:03 smithi196 ceph-mon[76349]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:05 smithi184 ceph-mon[100622]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:05 smithi184 ceph-mon[102844]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:05 smithi196 ceph-mon[76349]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:07 smithi184 ceph-mon[100622]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:07 smithi184 ceph-mon[102844]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:07 smithi196 ceph-mon[76349]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:08.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:09.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:09 smithi196 ceph-mon[76349]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:09.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[102844]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[100622]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[102844]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[100622]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:11.101Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:11.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:11.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:11 smithi196 ceph-mon[76349]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:37:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:13 smithi184 ceph-mon[100622]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:13 smithi184 ceph-mon[102844]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:13 smithi196 ceph-mon[76349]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:15 smithi184 ceph-mon[100622]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:15 smithi184 ceph-mon[102844]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:15 smithi196 ceph-mon[76349]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:17 smithi184 ceph-mon[100622]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:17 smithi184 ceph-mon[102844]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:17 smithi196 ceph-mon[76349]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:19 smithi196 ceph-mon[76349]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:19.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:19 smithi184 ceph-mon[100622]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:19.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:19 smithi184 ceph-mon[102844]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:21 smithi184 ceph-mon[100622]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:21 smithi184 ceph-mon[102844]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:21.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:21.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:21.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:21 smithi196 ceph-mon[76349]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:22] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:37:22.469 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:37:22.993 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:37:22.993 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 3h - - 2024-09-20T23:37:22.993 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 3h - - 2024-09-20T23:37:22.993 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 14s ago 3h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 3h 393M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 3h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 3h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:37:22.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 3h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:37:22.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:37:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:23 smithi184 ceph-mon[100622]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:23 smithi184 ceph-mon[102844]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:37:23.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:37:23.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:37:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:23 smithi196 ceph-mon[76349]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:37:23.577 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:37:23.578 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:37:23.578 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:37:23.578 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:37:23.887 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:37:23.887 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:23.887 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:23.887 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:23.887 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[100622]: from='client.69263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[100622]: from='client.59307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[100622]: from='client.59313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3371248535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1658488419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[102844]: from='client.69263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[102844]: from='client.59307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[102844]: from='client.59313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3371248535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1658488419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:24 smithi196 ceph-mon[76349]: from='client.69263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:24 smithi196 ceph-mon[76349]: from='client.59307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:24 smithi196 ceph-mon[76349]: from='client.59313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3371248535' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1658488419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:25 smithi196 ceph-mon[76349]: from='client.69287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:25 smithi196 ceph-mon[76349]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:25 smithi184 ceph-mon[100622]: from='client.69287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:25 smithi184 ceph-mon[100622]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:25 smithi184 ceph-mon[102844]: from='client.69287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:25 smithi184 ceph-mon[102844]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:27 smithi196 ceph-mon[76349]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:27 smithi184 ceph-mon[100622]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:27 smithi184 ceph-mon[102844]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:29 smithi196 ceph-mon[76349]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[100622]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[102844]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:31 smithi196 ceph-mon[76349]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[100622]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[102844]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:31.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:31.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:31.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:32] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:37:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:33 smithi196 ceph-mon[76349]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:33 smithi184 ceph-mon[100622]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:33 smithi184 ceph-mon[102844]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:35 smithi196 ceph-mon[76349]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:35 smithi184 ceph-mon[100622]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:35 smithi184 ceph-mon[102844]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:37 smithi196 ceph-mon[76349]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:37 smithi184 ceph-mon[100622]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:37 smithi184 ceph-mon[102844]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:39 smithi184 ceph-mon[102844]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:39 smithi184 ceph-mon[100622]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:39 smithi196 ceph-mon[76349]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:41.250 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:41 smithi184 ceph-mon[100622]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:41.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:41 smithi184 ceph-mon[102844]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:41.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:41.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:41.250 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:41.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:41 smithi196 ceph-mon[76349]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:42.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:42] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:37:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:43 smithi184 ceph-mon[100622]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:43 smithi184 ceph-mon[102844]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:43 smithi196 ceph-mon[76349]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:45 smithi184 ceph-mon[100622]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:45 smithi184 ceph-mon[102844]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:45 smithi196 ceph-mon[76349]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:47 smithi184 ceph-mon[100622]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:47 smithi184 ceph-mon[102844]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:47 smithi196 ceph-mon[76349]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[100622]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[102844]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:49 smithi196 ceph-mon[76349]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:51 smithi196 ceph-mon[76349]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:51.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.449 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[100622]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[102844]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:37:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:51.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:51.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:37:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:37:51.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:37:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:37:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:37:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:53 smithi184 ceph-mon[100622]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:53 smithi184 ceph-mon[102844]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:53 smithi196 ceph-mon[76349]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:54.183 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 3h - - 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 3h - - 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 4s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 3h 393M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:37:54.712 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 3h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 4s ago 3h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 3h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:37:54.713 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:37:54.714 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:37:54.714 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:37:55.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:37:55.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:37:55.296 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:37:55.297 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:37:55.297 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:37:55.297 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:37:55.297 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:37:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[102844]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[102844]: from='client.69299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3466366086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[100622]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[100622]: from='client.69299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3466366086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:55.606 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:37:55.606 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:55.606 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:55.606 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:55.606 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:55 smithi196 ceph-mon[76349]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:55 smithi196 ceph-mon[76349]: from='client.69299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3466366086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[102844]: from='client.69305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[102844]: from='client.69311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[102844]: from='client.59361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2866705248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[100622]: from='client.69305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[100622]: from='client.69311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[100622]: from='client.59361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2866705248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:56 smithi196 ceph-mon[76349]: from='client.69305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:56 smithi196 ceph-mon[76349]: from='client.69311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:56 smithi196 ceph-mon[76349]: from='client.59361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2866705248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:57 smithi184 ceph-mon[100622]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:57 smithi184 ceph-mon[102844]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:57 smithi196 ceph-mon[76349]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:59 smithi184 ceph-mon[100622]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:59 smithi184 ceph-mon[102844]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:59 smithi196 ceph-mon[76349]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:01.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:01.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:01.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:01.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:01.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:01.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:01.517 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:01 smithi184 ceph-mon[102844]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:01.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:01 smithi184 ceph-mon[100622]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:01 smithi196 ceph-mon[76349]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:38:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:03 smithi184 ceph-mon[100622]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:03 smithi184 ceph-mon[102844]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:03 smithi196 ceph-mon[76349]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:05 smithi184 ceph-mon[100622]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:05 smithi184 ceph-mon[102844]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:05 smithi196 ceph-mon[76349]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:07 smithi196 ceph-mon[76349]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:07 smithi184 ceph-mon[100622]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:07 smithi184 ceph-mon[102844]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:09.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:09 smithi196 ceph-mon[76349]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:09.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[100622]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[102844]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.402 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:11.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:11.402 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:11.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:11.402 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:11.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:11.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:11 smithi196 ceph-mon[76349]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:11.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[100622]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[102844]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:38:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:13 smithi184 ceph-mon[100622]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:13 smithi184 ceph-mon[102844]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:13 smithi196 ceph-mon[76349]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:38:13.813087101Z level=info msg="Completed cleanup jobs" duration=13.048481ms 2024-09-20T23:38:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:15 smithi184 ceph-mon[100622]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:15 smithi184 ceph-mon[102844]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:15 smithi196 ceph-mon[76349]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:17 smithi184 ceph-mon[100622]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:17 smithi184 ceph-mon[102844]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:17 smithi196 ceph-mon[76349]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:19 smithi184 ceph-mon[100622]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:19 smithi184 ceph-mon[102844]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:19 smithi196 ceph-mon[76349]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:20.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:20 smithi184 ceph-mon[100622]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:20.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:20 smithi184 ceph-mon[102844]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:20 smithi196 ceph-mon[76349]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:21.422 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:21.105Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:21.422 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:21.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:21.423 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:21.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:22] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:38:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:23 smithi184 ceph-mon[100622]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:23 smithi184 ceph-mon[102844]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:23 smithi196 ceph-mon[76349]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:25 smithi184 ceph-mon[100622]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:25 smithi184 ceph-mon[102844]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:25 smithi196 ceph-mon[76349]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:25.903 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:38:26.432 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:38:26.432 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 3h - - 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 17s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 17s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 17s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 17s ago 3h 394M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 17s ago 3h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 17s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 17s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:38:26.433 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 17s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 17s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 17s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 16s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 16s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 16s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 16s ago 3h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 17s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:38:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 16s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:38:26.741 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:38:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:38:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:38:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:38:26.742 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:38:27.010 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:38:27.011 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:38:27.011 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:38:27.011 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:38:27.011 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:38:27.011 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[102844]: from='client.69335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[102844]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[102844]: from='client.69341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[102844]: from='client.59385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/593276461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[100622]: from='client.69335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[100622]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[100622]: from='client.69341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[100622]: from='client.59385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/593276461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:27.319 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:38:27.320 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:27.320 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:27.320 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:27.320 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:27 smithi196 ceph-mon[76349]: from='client.69335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:27 smithi196 ceph-mon[76349]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:27 smithi196 ceph-mon[76349]: from='client.69341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:27 smithi196 ceph-mon[76349]: from='client.59385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/593276461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:28 smithi196 ceph-mon[76349]: from='client.59397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3504612146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[100622]: from='client.59397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3504612146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[102844]: from='client.59397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3504612146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[100622]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[102844]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:29 smithi196 ceph-mon[76349]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:31 smithi196 ceph-mon[76349]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[100622]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[102844]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:31.105Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:31.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:31.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:38:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:33 smithi196 ceph-mon[76349]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:33.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:33 smithi184 ceph-mon[100622]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:33.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:33 smithi184 ceph-mon[102844]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:35 smithi196 ceph-mon[76349]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:35 smithi184 ceph-mon[102844]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:35 smithi184 ceph-mon[100622]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:37 smithi184 ceph-mon[100622]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:37 smithi184 ceph-mon[102844]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:37 smithi196 ceph-mon[76349]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:39 smithi196 ceph-mon[76349]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:39 smithi184 ceph-mon[100622]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:39 smithi184 ceph-mon[102844]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:41 smithi184 ceph-mon[100622]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:41 smithi184 ceph-mon[102844]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:41.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:41.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:41.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:41.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:41 smithi196 ceph-mon[76349]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:38:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:43 smithi184 ceph-mon[100622]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:43 smithi184 ceph-mon[102844]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:43 smithi196 ceph-mon[76349]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:45 smithi196 ceph-mon[76349]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:45 smithi184 ceph-mon[100622]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:45 smithi184 ceph-mon[102844]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:47 smithi196 ceph-mon[76349]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:47 smithi184 ceph-mon[102844]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:47 smithi184 ceph-mon[100622]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:49 smithi196 ceph-mon[76349]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:49 smithi184 ceph-mon[100622]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:49 smithi184 ceph-mon[102844]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:51 smithi196 ceph-mon[76349]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[100622]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[102844]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:38:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:51.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:51.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:38:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:38:51.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:38:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:38:52] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:38:53.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:53 smithi196 ceph-mon[76349]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:53 smithi184 ceph-mon[100622]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:53 smithi184 ceph-mon[102844]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:55 smithi184 ceph-mon[100622]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:55 smithi184 ceph-mon[102844]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:55 smithi196 ceph-mon[76349]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:57 smithi184 ceph-mon[100622]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:57 smithi184 ceph-mon[102844]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:57.616 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:38:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:57 smithi196 ceph-mon[76349]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 3h - - 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 7s ago 3h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 9s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 9s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 9s ago 3h 395M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 3h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 9s ago 3h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:38:58.146 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 7s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 9s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 9s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 9s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 9s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 3h 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:38:58.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:38:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:38:58.457 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:38:58.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:58.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:58.725 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:38:58.725 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:38:58.725 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:38:58.726 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:38:59.055 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:38:59.055 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:59.055 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:59.055 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:59.056 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='client.69371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='client.69377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='client.59421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1805848229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1064949268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='client.69371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='client.69377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='client.59421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1805848229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:38:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1064949268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='client.69371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='client.69377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='client.59421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1805848229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:38:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1064949268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:00 smithi184 ceph-mon[100622]: from='client.69395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:00 smithi184 ceph-mon[102844]: from='client.69395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:00 smithi196 ceph-mon[76349]: from='client.69395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:01.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:01 smithi184 ceph-mon[100622]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:01.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:01 smithi184 ceph-mon[102844]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:01.440 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:01.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:01.440 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:01.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:01.440 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:01.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:01 smithi196 ceph-mon[76349]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:02.289 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:02] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:39:03.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:03 smithi184 ceph-mon[102844]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:03.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:03 smithi184 ceph-mon[100622]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:03 smithi196 ceph-mon[76349]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:05 smithi184 ceph-mon[100622]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:05 smithi184 ceph-mon[102844]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:05 smithi196 ceph-mon[76349]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:07 smithi184 ceph-mon[100622]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:07 smithi184 ceph-mon[102844]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:07 smithi196 ceph-mon[76349]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:09 smithi184 ceph-mon[100622]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:09 smithi184 ceph-mon[102844]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:09 smithi196 ceph-mon[76349]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[100622]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[102844]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:11.108Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:11.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:11.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:11.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:11 smithi196 ceph-mon[76349]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:11.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:11.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:12] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:39:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:13 smithi184 ceph-mon[100622]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:13 smithi184 ceph-mon[102844]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:13 smithi196 ceph-mon[76349]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:15 smithi184 ceph-mon[100622]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:15 smithi184 ceph-mon[102844]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:15 smithi196 ceph-mon[76349]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:17 smithi184 ceph-mon[102844]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:17 smithi184 ceph-mon[100622]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:17 smithi196 ceph-mon[76349]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:19 smithi196 ceph-mon[76349]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:19 smithi184 ceph-mon[102844]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:19 smithi184 ceph-mon[100622]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:21 smithi184 ceph-mon[102844]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:21 smithi184 ceph-mon[100622]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:21.109Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:21.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:21.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:21 smithi196 ceph-mon[76349]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:22.215 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:39:23.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:23 smithi184 ceph-mon[100622]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:23.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:23 smithi184 ceph-mon[102844]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:23 smithi196 ceph-mon[76349]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:25 smithi184 ceph-mon[100622]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:25 smithi184 ceph-mon[102844]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:25.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:25 smithi196 ceph-mon[76349]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:27 smithi196 ceph-mon[76349]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:27 smithi184 ceph-mon[100622]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:27 smithi184 ceph-mon[102844]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:39:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:29 smithi196 ceph-mon[76349]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:29 smithi184 ceph-mon[100622]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:29 smithi184 ceph-mon[102844]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 3h - - 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 0s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 3h 397M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 19s ago 3h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:39:29.894 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 3h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 0s ago 3h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 19s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 19s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 19s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 19s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 19s ago 3h 72.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:39:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:39:29.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 19s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:39:30.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:39:30.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:39:30.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:39:30.203 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:39:30.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[100622]: from='client.69407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:30.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1046673864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:30.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[102844]: from='client.69407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:30.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1046673864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:30.784 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:39:30.784 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:30.784 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:30.784 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:30.784 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:30 smithi196 ceph-mon[76349]: from='client.69407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1046673864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:31.409 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:31.109Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:31.409 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:31.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:31.409 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:31.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='client.69413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='client.59457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2495226960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='client.69413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='client.59457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2495226960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='client.69413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='client.59457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2495226960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:39:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:32 smithi196 ceph-mon[76349]: from='client.69431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:32 smithi184 ceph-mon[102844]: from='client.69431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:32 smithi184 ceph-mon[100622]: from='client.69431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:33.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:33 smithi196 ceph-mon[76349]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:33 smithi184 ceph-mon[102844]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:33 smithi184 ceph-mon[100622]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:35 smithi184 ceph-mon[102844]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:35 smithi184 ceph-mon[100622]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:35 smithi196 ceph-mon[76349]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:37 smithi184 ceph-mon[102844]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:37 smithi184 ceph-mon[100622]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:37 smithi196 ceph-mon[76349]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:39 smithi184 ceph-mon[102844]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:39 smithi184 ceph-mon[100622]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:39 smithi196 ceph-mon[76349]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:41.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:41.110Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:41.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:41.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:41.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:41.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:41 smithi184 ceph-mon[102844]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:41 smithi184 ceph-mon[100622]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:41 smithi196 ceph-mon[76349]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:39:43.676 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:43 smithi184 ceph-mon[100622]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:43.676 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:43 smithi184 ceph-mon[102844]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:43 smithi196 ceph-mon[76349]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:45 smithi184 ceph-mon[102844]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:45 smithi184 ceph-mon[100622]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:45 smithi196 ceph-mon[76349]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:46 smithi184 ceph-mon[102844]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:46 smithi184 ceph-mon[100622]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:46 smithi196 ceph-mon[76349]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:48 smithi184 ceph-mon[100622]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:48 smithi184 ceph-mon[102844]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:48 smithi196 ceph-mon[76349]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[100622]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:50 smithi184 ceph-mon[102844]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:50 smithi196 ceph-mon[76349]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:51.111Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:51.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:39:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:39:51.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:39:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:39:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:39:53.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:53 smithi196 ceph-mon[76349]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:53 smithi184 ceph-mon[102844]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:53 smithi184 ceph-mon[100622]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:55 smithi196 ceph-mon[76349]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:55 smithi184 ceph-mon[100622]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:55 smithi184 ceph-mon[102844]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:57 smithi184 ceph-mon[100622]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:57 smithi184 ceph-mon[102844]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:57 smithi196 ceph-mon[76349]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:59 smithi196 ceph-mon[76349]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:59 smithi184 ceph-mon[100622]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:59 smithi184 ceph-mon[102844]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:39:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:01.079 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:40:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:01 smithi196 ceph-mon[76349]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:01 smithi184 ceph-mon[100622]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:01 smithi184 ceph-mon[102844]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:01.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:01.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:01.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 3h - - 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 3h - - 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 12s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 10s ago 3h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 12s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 10s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 12s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 12s ago 3h 397M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 10s ago 3h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 12s ago 3h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:40:01.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 12s ago 3h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 10s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 12s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 12s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 12s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 12s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 10s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 10s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 10s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 3h 68.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 12s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:40:01.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 10s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:40:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:40:01.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:40:02.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[100622]: from='client.69443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[100622]: from='client.69449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2136871216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:02.194 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:40:02.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[102844]: from='client.69443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[102844]: from='client.69449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2136871216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:02.195 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:40:02.195 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:40:02.195 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:40:02.196 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:40:02.197 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:40:02.197 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:40:02.197 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:40:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:02 smithi196 ceph-mon[76349]: from='client.69443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:02 smithi196 ceph-mon[76349]: from='client.69449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2136871216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:02.512 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:02.512 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:02.513 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:02.513 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:02.513 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:03.366 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[100622]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[100622]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[100622]: from='client.69467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2550266185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[102844]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[102844]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[102844]: from='client.69467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2550266185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:03 smithi196 ceph-mon[76349]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:03 smithi196 ceph-mon[76349]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:03 smithi196 ceph-mon[76349]: from='client.69467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2550266185' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:05.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:05 smithi184 ceph-mon[100622]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:05.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:05 smithi184 ceph-mon[102844]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:05 smithi196 ceph-mon[76349]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:07 smithi184 ceph-mon[100622]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:07 smithi184 ceph-mon[102844]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:07 smithi196 ceph-mon[76349]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:09 smithi196 ceph-mon[76349]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:09 smithi184 ceph-mon[100622]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:09 smithi184 ceph-mon[102844]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[100622]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:10 smithi184 ceph-mon[102844]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:10 smithi196 ceph-mon[76349]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:11.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:11.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:11.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:40:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:13 smithi184 ceph-mon[100622]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:13 smithi184 ceph-mon[102844]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:13.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:13 smithi196 ceph-mon[76349]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:14.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:15 smithi184 ceph-mon[100622]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:15 smithi184 ceph-mon[102844]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:15 smithi196 ceph-mon[76349]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:17 smithi184 ceph-mon[102844]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:17 smithi184 ceph-mon[100622]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:17 smithi196 ceph-mon[76349]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:19 smithi184 ceph-mon[102844]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:19 smithi184 ceph-mon[100622]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:19 smithi196 ceph-mon[76349]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:21 smithi184 ceph-mon[100622]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:21 smithi184 ceph-mon[102844]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:21.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:21.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:21.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:21 smithi196 ceph-mon[76349]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:40:23.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:23 smithi196 ceph-mon[76349]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:23 smithi184 ceph-mon[100622]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:23 smithi184 ceph-mon[102844]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:25.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:25 smithi184 ceph-mon[100622]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:25.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:25 smithi184 ceph-mon[102844]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:25 smithi196 ceph-mon[76349]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:27 smithi184 ceph-mon[100622]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:27 smithi184 ceph-mon[102844]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:27 smithi196 ceph-mon[76349]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:29 smithi184 ceph-mon[102844]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:29 smithi184 ceph-mon[100622]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:29 smithi196 ceph-mon[76349]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[100622]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:30 smithi184 ceph-mon[102844]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:30 smithi196 ceph-mon[76349]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:31.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:31.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:31.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:40:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:32.852 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 3h - - 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 3h - - 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 3s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:40:33.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 3s ago 3h 399M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 2s ago 3h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 3s ago 3h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 3s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 3s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 3s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 3s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 3s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 2s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 2s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 2s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 2s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:40:33.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:40:33.401 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 3s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:40:33.401 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 2s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:40:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:33 smithi196 ceph-mon[76349]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:33.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:33 smithi184 ceph-mon[100622]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:33.717 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:40:33.717 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:40:33.717 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:40:33.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:33.717 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:40:33.718 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:40:33.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:33 smithi184 ceph-mon[102844]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:40:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:40:33.993 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:40:34.301 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:34.301 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:34.302 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:34.302 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:34.302 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:34.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:34 smithi196 ceph-mon[76349]: from='client.59517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:34 smithi196 ceph-mon[76349]: from='client.69485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:34 smithi196 ceph-mon[76349]: from='client.59529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1954733945' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:34.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/936347764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[102844]: from='client.59517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[102844]: from='client.69485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[102844]: from='client.59529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1954733945' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/936347764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[100622]: from='client.59517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[100622]: from='client.69485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[100622]: from='client.59529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1954733945' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/936347764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:35.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:35 smithi196 ceph-mon[76349]: from='client.59541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:35.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:35 smithi196 ceph-mon[76349]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:35 smithi184 ceph-mon[102844]: from='client.59541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:35 smithi184 ceph-mon[102844]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:35 smithi184 ceph-mon[100622]: from='client.59541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:35 smithi184 ceph-mon[100622]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:37 smithi196 ceph-mon[76349]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:37 smithi184 ceph-mon[102844]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:37 smithi184 ceph-mon[100622]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:39 smithi196 ceph-mon[76349]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:39 smithi184 ceph-mon[102844]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:39 smithi184 ceph-mon[100622]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:41 smithi184 ceph-mon[100622]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:41 smithi184 ceph-mon[102844]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:41.114Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:41.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:41.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:41 smithi196 ceph-mon[76349]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:40:43.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:43 smithi184 ceph-mon[100622]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:43.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:43 smithi184 ceph-mon[102844]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:43 smithi196 ceph-mon[76349]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:44.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:44.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:45 smithi196 ceph-mon[76349]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:45.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:45 smithi184 ceph-mon[100622]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:45.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:45 smithi184 ceph-mon[102844]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:47 smithi196 ceph-mon[76349]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:47 smithi184 ceph-mon[100622]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:47 smithi184 ceph-mon[102844]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:49 smithi196 ceph-mon[76349]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:49 smithi184 ceph-mon[100622]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:49 smithi184 ceph-mon[102844]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[100622]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:50 smithi184 ceph-mon[102844]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:51.114Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:51.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:40:51.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:40:51.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:40:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:50 smithi196 ceph-mon[76349]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:40:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:40:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:40:53.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:53 smithi196 ceph-mon[76349]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:53 smithi184 ceph-mon[102844]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:53 smithi184 ceph-mon[100622]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:55.793 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:55 smithi196 ceph-mon[76349]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:55 smithi184 ceph-mon[100622]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:55 smithi184 ceph-mon[102844]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:56 smithi184 ceph-mon[100622]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:56 smithi184 ceph-mon[102844]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:56 smithi196 ceph-mon[76349]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:58 smithi184 ceph-mon[100622]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:58 smithi184 ceph-mon[102844]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:58 smithi196 ceph-mon[76349]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:00 smithi184 ceph-mon[100622]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:00 smithi184 ceph-mon[102844]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:00 smithi196 ceph-mon[76349]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:01.115Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:01.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:01.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:41:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:03 smithi196 ceph-mon[76349]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:03.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:03 smithi184 ceph-mon[100622]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:03.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:03 smithi184 ceph-mon[102844]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:04.593 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 3h - - 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 3h - - 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 15s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 13s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 15s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 15s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:41:05.118 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 15s ago 3h 399M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 13s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 15s ago 3h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 15s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 13s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 15s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 15s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 15s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 15s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 13s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 13s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 13s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 13s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:41:05.119 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 3h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:41:05.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:41:05.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:41:05.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 15s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:41:05.120 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 13s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:41:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:05 smithi196 ceph-mon[76349]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:05.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:05 smithi184 ceph-mon[100622]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:05.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:05 smithi184 ceph-mon[102844]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:05.430 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:41:05.431 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:41:05.710 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:41:05.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:41:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:41:06.026 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:06.026 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[100622]: from='client.59553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[100622]: from='client.69521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[100622]: from='client.69527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2824740837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3308625650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[102844]: from='client.59553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[102844]: from='client.69521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[102844]: from='client.69527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2824740837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3308625650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:06 smithi196 ceph-mon[76349]: from='client.59553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:06 smithi196 ceph-mon[76349]: from='client.69521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:06 smithi196 ceph-mon[76349]: from='client.69527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2824740837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3308625650' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:07 smithi184 ceph-mon[100622]: from='client.69539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:07 smithi184 ceph-mon[100622]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:07 smithi184 ceph-mon[102844]: from='client.69539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:07 smithi184 ceph-mon[102844]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:07 smithi196 ceph-mon[76349]: from='client.69539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:07 smithi196 ceph-mon[76349]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:09 smithi196 ceph-mon[76349]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:09 smithi184 ceph-mon[100622]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:09 smithi184 ceph-mon[102844]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[100622]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[102844]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:11.117Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:11.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:11.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:11.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:11 smithi196 ceph-mon[76349]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:41:12.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:12 smithi196 ceph-mon[76349]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[102844]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:12.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:12 smithi184 ceph-mon[100622]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:15 smithi184 ceph-mon[100622]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:15 smithi184 ceph-mon[102844]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:14 smithi196 ceph-mon[76349]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:17.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:17 smithi196 ceph-mon[76349]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:17 smithi184 ceph-mon[100622]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:17 smithi184 ceph-mon[102844]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:19 smithi196 ceph-mon[76349]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:19 smithi184 ceph-mon[100622]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:19 smithi184 ceph-mon[102844]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:41:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:21 smithi196 ceph-mon[76349]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:21 smithi184 ceph-mon[100622]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:21 smithi184 ceph-mon[102844]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:21.117Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:21.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:21.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:41:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:23 smithi196 ceph-mon[76349]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:23 smithi184 ceph-mon[100622]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:23 smithi184 ceph-mon[102844]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:25 smithi196 ceph-mon[76349]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:25 smithi184 ceph-mon[100622]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:25 smithi184 ceph-mon[102844]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:27.241 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:27 smithi184 ceph-mon[100622]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:27.241 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:27 smithi184 ceph-mon[102844]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:27 smithi196 ceph-mon[76349]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:28.394 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:28.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:29 smithi196 ceph-mon[76349]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:29 smithi184 ceph-mon[100622]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:29 smithi184 ceph-mon[102844]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[100622]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[102844]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:31.118Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:31.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:31.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:31 smithi196 ceph-mon[76349]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:41:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[100622]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:32 smithi184 ceph-mon[102844]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:32 smithi196 ceph-mon[76349]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:35 smithi196 ceph-mon[76349]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:35 smithi184 ceph-mon[100622]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:35 smithi184 ceph-mon[102844]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:36.322 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:41:36.848 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:41:36.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 3h - - 2024-09-20T23:41:36.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 3h - - 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 6s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 6s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 6s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 6s ago 3h 401M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 3h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 6s ago 3h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 6s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 6s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 6s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 6s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 6s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 5s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:41:36.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 5s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 5s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 5s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 69.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 6s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:41:36.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 5s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:41:37.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:41:37.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:41:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:37 smithi196 ceph-mon[76349]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:37 smithi196 ceph-mon[76349]: from='client.69551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:41:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:41:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:41:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:41:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:41:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:41:37.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:37 smithi184 ceph-mon[100622]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:37.430 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:37 smithi184 ceph-mon[100622]: from='client.69551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:37.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:37 smithi184 ceph-mon[102844]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:37.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:37 smithi184 ceph-mon[102844]: from='client.69551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:37.740 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:41:37.740 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:37.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:37.740 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:37.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.69557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.59601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1761906072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.59613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3511899436' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.69557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.59601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1761906072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.59613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3511899436' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.69557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.59601 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1761906072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.59613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3511899436' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:39 smithi196 ceph-mon[76349]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:39 smithi184 ceph-mon[100622]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:39 smithi184 ceph-mon[102844]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:41 smithi196 ceph-mon[76349]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:41 smithi184 ceph-mon[100622]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:41 smithi184 ceph-mon[102844]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:41.119Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:41.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:41.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:41:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:43 smithi196 ceph-mon[76349]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:43 smithi184 ceph-mon[100622]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:43 smithi184 ceph-mon[102844]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:45 smithi196 ceph-mon[76349]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:45 smithi184 ceph-mon[100622]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:45 smithi184 ceph-mon[102844]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:47 smithi196 ceph-mon[76349]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:47 smithi184 ceph-mon[100622]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:47 smithi184 ceph-mon[102844]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:49 smithi196 ceph-mon[76349]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:49 smithi184 ceph-mon[100622]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:49 smithi184 ceph-mon[102844]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[100622]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:51.119Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:51.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:41:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:41:51.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:41:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[102844]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:51 smithi196 ceph-mon[76349]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:41:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:52 smithi196 ceph-mon[76349]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[102844]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:41:53.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:52 smithi184 ceph-mon[100622]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:55 smithi196 ceph-mon[76349]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:55 smithi184 ceph-mon[102844]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:55 smithi184 ceph-mon[100622]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:57 smithi196 ceph-mon[76349]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:57 smithi184 ceph-mon[100622]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:57 smithi184 ceph-mon[102844]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:59.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:59 smithi196 ceph-mon[76349]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:59 smithi184 ceph-mon[100622]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:59 smithi184 ceph-mon[102844]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:01 smithi196 ceph-mon[76349]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:01 smithi184 ceph-mon[100622]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:01 smithi184 ceph-mon[102844]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:01.120Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:01.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:01.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:42:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:03 smithi196 ceph-mon[76349]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:03.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:03 smithi184 ceph-mon[102844]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:03.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:03 smithi184 ceph-mon[100622]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:05 smithi196 ceph-mon[76349]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:05 smithi184 ceph-mon[100622]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:05 smithi184 ceph-mon[102844]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:07 smithi184 ceph-mon[100622]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:07 smithi184 ceph-mon[102844]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:07 smithi196 ceph-mon[76349]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:08.041 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:42:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:08.621 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:42:08.621 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 3h - - 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 3h - - 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 18s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 16s ago 3h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 18s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 16s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 18s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 3h 401M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 3h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 18s ago 3h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 16s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 18s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 18s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:42:08.622 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 18s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 16s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 16s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 16s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 16s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 16s ago 3h 69.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:42:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 16s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:42:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:42:08.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:42:08.940 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:42:09.221 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:42:09.222 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:42:09.222 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:42:09.222 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:42:09.222 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:42:09.222 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[102844]: from='client.59625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[102844]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[102844]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2274651636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[100622]: from='client.59625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[100622]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:09.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[100622]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2274651636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:42:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:09 smithi196 ceph-mon[76349]: from='client.59625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:09 smithi196 ceph-mon[76349]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:09 smithi196 ceph-mon[76349]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2274651636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[100622]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[100622]: from='client.69611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2654707141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[102844]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[102844]: from='client.69611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2654707141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:10 smithi196 ceph-mon[76349]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:10 smithi196 ceph-mon[76349]: from='client.69611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2654707141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:11.423 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:11.120Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:11.423 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:11.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:11.423 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:11.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[100622]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[102844]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:11 smithi196 ceph-mon[76349]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[100622]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:12 smithi184 ceph-mon[102844]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:12 smithi196 ceph-mon[76349]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:14 smithi184 ceph-mon[100622]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:14 smithi184 ceph-mon[102844]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:14 smithi196 ceph-mon[76349]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:17 smithi184 ceph-mon[100622]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:17 smithi184 ceph-mon[102844]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:17 smithi196 ceph-mon[76349]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:19.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:19 smithi196 ceph-mon[76349]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:19 smithi184 ceph-mon[102844]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:19 smithi184 ceph-mon[100622]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:21 smithi196 ceph-mon[76349]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:21 smithi184 ceph-mon[100622]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:21 smithi184 ceph-mon[102844]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:21.121Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:21.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:21.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:42:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:23 smithi196 ceph-mon[76349]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:23 smithi184 ceph-mon[102844]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:23 smithi184 ceph-mon[100622]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:25 smithi184 ceph-mon[100622]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:25 smithi184 ceph-mon[102844]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:25 smithi196 ceph-mon[76349]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:27 smithi184 ceph-mon[100622]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:27 smithi184 ceph-mon[102844]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:27 smithi196 ceph-mon[76349]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:28.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:28.438 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:29.451 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:29 smithi184 ceph-mon[100622]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:29.451 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:29.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:29 smithi184 ceph-mon[102844]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:29.452 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:29 smithi196 ceph-mon[76349]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[100622]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[102844]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:31.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:31.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:31.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:31.550 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:31.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:31 smithi196 ceph-mon[76349]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:42:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:33 smithi196 ceph-mon[76349]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[102844]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[100622]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:35 smithi196 ceph-mon[76349]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:35 smithi184 ceph-mon[100622]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:35 smithi184 ceph-mon[102844]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:37 smithi184 ceph-mon[100622]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:37 smithi184 ceph-mon[102844]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:37 smithi196 ceph-mon[76349]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:38.368 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:39.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:39 smithi196 ceph-mon[76349]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:39 smithi184 ceph-mon[100622]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:39 smithi184 ceph-mon[102844]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.838 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 3h - - 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 3h - - 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 9s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 8s ago 3h 91.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:42:40.368 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 9s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 8s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 9s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 9s ago 4h 402M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 3h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 9s ago 3h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 9s ago 3h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 8s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 9s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 9s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 9s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 9s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 3h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 8s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 8s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:42:40.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 8s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:42:40.370 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 8s ago 3h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:42:40.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:42:40.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:42:40.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 9s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:42:40.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 8s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:42:40.678 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:42:40.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:42:40.680 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:42:40.948 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:42:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:42:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:42:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:42:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:42:40.949 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:42:41.260 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:42:41.260 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:41.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:41.260 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:41.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:41.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:41.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:41.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:41.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:41.261 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:41.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:41.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[100622]: from='client.69623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[100622]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:41.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[100622]: from='client.59667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[100622]: from='client.69635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3646293148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:41.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[102844]: from='client.69623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[102844]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:41.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[102844]: from='client.59667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[102844]: from='client.69635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3646293148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:41 smithi196 ceph-mon[76349]: from='client.69623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:41 smithi196 ceph-mon[76349]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:41 smithi196 ceph-mon[76349]: from='client.59667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:41 smithi196 ceph-mon[76349]: from='client.69635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3646293148' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:42 smithi184 ceph-mon[100622]: from='client.69647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1011682031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:42:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:42 smithi184 ceph-mon[102844]: from='client.69647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1011682031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:42 smithi196 ceph-mon[76349]: from='client.69647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1011682031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:43 smithi196 ceph-mon[76349]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:43 smithi184 ceph-mon[100622]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:43 smithi184 ceph-mon[102844]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:45 smithi196 ceph-mon[76349]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:45 smithi184 ceph-mon[100622]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:45 smithi184 ceph-mon[102844]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:47 smithi196 ceph-mon[76349]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:47 smithi184 ceph-mon[100622]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:47 smithi184 ceph-mon[102844]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:49 smithi184 ceph-mon[100622]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:49 smithi184 ceph-mon[102844]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:49 smithi196 ceph-mon[76349]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:51.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[100622]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:51.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[102844]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.274 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:51.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:51.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:51.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:42:51.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:42:51.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:42:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:51 smithi196 ceph-mon[76349]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:42:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:42:53.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:53 smithi196 ceph-mon[76349]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:53.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.542 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[100622]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[102844]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:42:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:55 smithi184 ceph-mon[100622]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:55 smithi184 ceph-mon[102844]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:55 smithi196 ceph-mon[76349]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:57 smithi184 ceph-mon[100622]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:57 smithi184 ceph-mon[102844]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:57 smithi196 ceph-mon[76349]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:59.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:59 smithi196 ceph-mon[76349]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:59.520 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:59 smithi184 ceph-mon[100622]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:59 smithi184 ceph-mon[102844]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:01.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:01 smithi196 ceph-mon[76349]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:01 smithi184 ceph-mon[100622]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:01 smithi184 ceph-mon[102844]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:01.123Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:01.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:01.124Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:43:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:03 smithi184 ceph-mon[100622]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:03 smithi184 ceph-mon[102844]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:03 smithi196 ceph-mon[76349]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:05 smithi184 ceph-mon[100622]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:05 smithi184 ceph-mon[102844]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:05 smithi196 ceph-mon[76349]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:07.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:07 smithi184 ceph-mon[100622]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:07.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:07 smithi184 ceph-mon[102844]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:07 smithi196 ceph-mon[76349]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:09 smithi184 ceph-mon[100622]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:09 smithi184 ceph-mon[102844]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:09 smithi196 ceph-mon[76349]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:11.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[100622]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:11.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.455 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:11.124Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:11.455 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:11.124Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:11.455 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:11.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:11.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[102844]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:11.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.624 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:43:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:11 smithi196 ceph-mon[76349]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:12.166 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:43:12.166 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 3h - - 2024-09-20T23:43:12.166 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 3h - - 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 1s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 19s ago 3h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 1s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 19s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 1s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 1s ago 4h 403M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 19s ago 3h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 1s ago 3h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 1s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 19s ago 3h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 1s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 1s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 1s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 19s ago 3h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:43:12.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 19s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 19s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 19s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 19s ago 3h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:43:12.168 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 19s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:43:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:12.495 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:43:12.496 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:43:12.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:43:12.766 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:43:12.766 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:43:12.766 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:43:12.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:43:13.075 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:43:13.076 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:13.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:13.076 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:13.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='client.59697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='client.69665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='client.69671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3811272656' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:13.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1609171733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='client.59697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='client.69665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='client.69671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3811272656' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1609171733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='client.59697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='client.69665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='client.69671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3811272656' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:13.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1609171733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:14 smithi196 ceph-mon[76349]: from='client.59721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:14 smithi184 ceph-mon[100622]: from='client.59721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:14 smithi184 ceph-mon[102844]: from='client.59721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:15 smithi196 ceph-mon[76349]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:15 smithi184 ceph-mon[100622]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:15 smithi184 ceph-mon[102844]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:17 smithi196 ceph-mon[76349]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:17 smithi184 ceph-mon[100622]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:17 smithi184 ceph-mon[102844]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:19.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:19 smithi196 ceph-mon[76349]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:19 smithi184 ceph-mon[100622]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:19 smithi184 ceph-mon[102844]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:21.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:21 smithi196 ceph-mon[76349]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:21 smithi184 ceph-mon[100622]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:21 smithi184 ceph-mon[102844]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:21.125Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:21.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:21.126Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:43:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:23 smithi196 ceph-mon[76349]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:23 smithi184 ceph-mon[100622]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:23 smithi184 ceph-mon[102844]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:25 smithi196 ceph-mon[76349]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:25 smithi184 ceph-mon[100622]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:25 smithi184 ceph-mon[102844]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:27 smithi196 ceph-mon[76349]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:27 smithi184 ceph-mon[100622]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:27 smithi184 ceph-mon[102844]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:29 smithi196 ceph-mon[76349]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:29 smithi184 ceph-mon[100622]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:29 smithi184 ceph-mon[102844]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:31.126Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:31.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:31.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:31 smithi196 ceph-mon[76349]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[100622]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:31.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[102844]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:31.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:32.020 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:43:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[100622]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[102844]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:33 smithi196 ceph-mon[76349]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:35 smithi184 ceph-mon[100622]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:35 smithi184 ceph-mon[102844]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:35 smithi196 ceph-mon[76349]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:37 smithi184 ceph-mon[100622]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:37 smithi184 ceph-mon[102844]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:37 smithi196 ceph-mon[76349]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:39 smithi184 ceph-mon[100622]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:39 smithi184 ceph-mon[102844]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:39.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:39 smithi196 ceph-mon[76349]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:40 smithi184 ceph-mon[100622]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:40 smithi184 ceph-mon[102844]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:40 smithi196 ceph-mon[76349]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:41.126Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:41.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:41.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:43:43.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:43 smithi196 ceph-mon[76349]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.374 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:43:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:43 smithi184 ceph-mon[100622]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:43 smithi184 ceph-mon[102844]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 12s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 11s ago 3h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 12s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 11s ago 3h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:43:43.906 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 12s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 12s ago 4h 403M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 11s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 12s ago 3h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 12s ago 3h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 11s ago 3h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 12s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 12s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 12s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 12s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 3h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:43:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 11s ago 3h 70.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:43:43.908 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:43:43.908 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:43:43.908 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 12s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:43:43.908 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:43:44.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:43:44.226 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:43:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:44 smithi196 ceph-mon[76349]: from='client.59733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:43:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:43:44.495 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:43:44.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:44 smithi184 ceph-mon[100622]: from='client.59733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:44.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.496 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:44 smithi184 ceph-mon[102844]: from='client.59733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:44.496 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.804 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:43:44.804 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:44.805 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:44.805 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:44.805 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[102844]: from='client.69701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[102844]: from='client.69707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[102844]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/134521894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/827118124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[100622]: from='client.69701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[100622]: from='client.69707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[100622]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/134521894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/827118124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:45 smithi196 ceph-mon[76349]: from='client.69701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:45 smithi196 ceph-mon[76349]: from='client.69707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:45 smithi196 ceph-mon[76349]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/134521894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/827118124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:46 smithi184 ceph-mon[100622]: from='client.59757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:46 smithi184 ceph-mon[102844]: from='client.59757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:46 smithi196 ceph-mon[76349]: from='client.59757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:47 smithi184 ceph-mon[100622]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:47 smithi184 ceph-mon[102844]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:47 smithi196 ceph-mon[76349]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:49 smithi184 ceph-mon[100622]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:49 smithi184 ceph-mon[102844]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:49 smithi196 ceph-mon[76349]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:51.335 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[100622]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[102844]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.336 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:51.128Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:51.337 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:51.128Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:43:51.337 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:43:51.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:43:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:51 smithi196 ceph-mon[76349]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:43:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[100622]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[102844]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:53.521 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:53 smithi196 ceph-mon[76349]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:53.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:53.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:43:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:55 smithi184 ceph-mon[100622]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:55 smithi184 ceph-mon[102844]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:55 smithi196 ceph-mon[76349]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:57 smithi184 ceph-mon[100622]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:57 smithi184 ceph-mon[102844]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:57 smithi196 ceph-mon[76349]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:59 smithi184 ceph-mon[100622]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:59 smithi184 ceph-mon[102844]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:59 smithi196 ceph-mon[76349]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:01 smithi196 ceph-mon[76349]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:01 smithi184 ceph-mon[100622]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:01 smithi184 ceph-mon[102844]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:01.129Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:01.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:01.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:02] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:44:03.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:03 smithi184 ceph-mon[100622]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:03.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:03 smithi184 ceph-mon[102844]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:03 smithi196 ceph-mon[76349]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:05 smithi184 ceph-mon[100622]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:05 smithi184 ceph-mon[102844]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:05 smithi196 ceph-mon[76349]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:07 smithi184 ceph-mon[100622]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:44:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:07 smithi184 ceph-mon[102844]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:44:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:07 smithi196 ceph-mon[76349]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:44:08.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:09 smithi184 ceph-mon[100622]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:44:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:09 smithi184 ceph-mon[102844]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:44:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:09 smithi196 ceph-mon[76349]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:44:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[100622]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:44:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[102844]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:44:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:11.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:11.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:11.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:11 smithi196 ceph-mon[76349]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:44:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:12] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[100622]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[102844]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:13 smithi196 ceph-mon[76349]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:44:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:14.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:14.509 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:15.101 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:44:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:15 smithi184 ceph-mon[102844]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:44:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:15 smithi184 ceph-mon[100622]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 4s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 2s ago 3h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 4s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 4s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 4s ago 4h 404M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 2s ago 4h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:44:15.638 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 4s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 4s ago 3h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 2s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 4s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 3h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 3h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 4s ago 3h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 2s ago 3h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 2s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 2s ago 3h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 2s ago 3h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 2s ago 3h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 2s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:44:15.639 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:44:15.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 2s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:44:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:15 smithi196 ceph-mon[76349]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:15.948 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:44:15.949 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:44:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:44:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:44:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:44:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:44:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:44:16.221 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:44:16.530 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:44:16.530 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:16.530 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:16.531 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:16.531 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[100622]: from='client.69731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[100622]: from='client.59775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2172658657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[102844]: from='client.69731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[102844]: from='client.59775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2172658657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:16 smithi196 ceph-mon[76349]: from='client.69731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:16 smithi196 ceph-mon[76349]: from='client.59775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2172658657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[100622]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[100622]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[100622]: from='client.59793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/733389370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[102844]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[102844]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:44:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[102844]: from='client.59793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/733389370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:17 smithi196 ceph-mon[76349]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:17 smithi196 ceph-mon[76349]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:17 smithi196 ceph-mon[76349]: from='client.59793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/733389370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:19 smithi184 ceph-mon[100622]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:44:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:19 smithi184 ceph-mon[102844]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:44:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:19 smithi196 ceph-mon[76349]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 308 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:44:21.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:21 smithi196 ceph-mon[76349]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:44:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:21 smithi184 ceph-mon[100622]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:44:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:21 smithi184 ceph-mon[102844]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:44:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:21.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:21.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:21.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:44:23.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:23 smithi196 ceph-mon[76349]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:23 smithi184 ceph-mon[100622]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:23 smithi184 ceph-mon[102844]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:25 smithi184 ceph-mon[100622]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:25 smithi184 ceph-mon[102844]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:25 smithi196 ceph-mon[76349]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T23:44:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:27 smithi184 ceph-mon[100622]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:44:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:27 smithi184 ceph-mon[102844]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:44:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:27 smithi196 ceph-mon[76349]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 312 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:44:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:29 smithi184 ceph-mon[100622]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:29 smithi184 ceph-mon[102844]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:29 smithi196 ceph-mon[76349]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:31 smithi184 ceph-mon[100622]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:44:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:31 smithi184 ceph-mon[102844]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:44:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:31.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:31.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:31.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:31 smithi196 ceph-mon[76349]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:44:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:44:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:32.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:33 smithi196 ceph-mon[76349]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:44:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[100622]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[102844]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 304 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:44:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:35.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:35 smithi184 ceph-mon[100622]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:44:35.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:35 smithi184 ceph-mon[102844]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:44:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:35 smithi196 ceph-mon[76349]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:44:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:37 smithi184 ceph-mon[100622]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:37 smithi184 ceph-mon[102844]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:37 smithi196 ceph-mon[76349]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:44:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:39 smithi196 ceph-mon[76349]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:44:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:39 smithi184 ceph-mon[100622]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:44:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:39 smithi184 ceph-mon[102844]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T23:44:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:41 smithi184 ceph-mon[100622]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T23:44:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:41 smithi184 ceph-mon[102844]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T23:44:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:41.132Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:41.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:41.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:41 smithi196 ceph-mon[76349]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T23:44:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:44:43.506 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:43 smithi196 ceph-mon[76349]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T23:44:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:43 smithi184 ceph-mon[100622]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T23:44:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:43 smithi184 ceph-mon[102844]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-20T23:44:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:45 smithi196 ceph-mon[76349]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T23:44:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:45 smithi184 ceph-mon[100622]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T23:44:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:45 smithi184 ceph-mon[102844]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-20T23:44:46.824 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 14s ago 3h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:44:47.351 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 4h 405M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 14s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 4h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 16s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 14s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 3h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 3h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 3h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:44:47.352 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 3h 80.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 3h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 3h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 14s ago 3h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:44:47.353 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:44:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:47 smithi184 ceph-mon[100622]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T23:44:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:47 smithi184 ceph-mon[102844]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T23:44:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:47 smithi196 ceph-mon[76349]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:44:47.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:44:47.666 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:44:47.938 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:44:47.939 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:44:47.939 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:44:47.939 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:44:47.939 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:44:48.248 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:44:48.249 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:48.249 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:48.249 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:48.249 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.59805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.59811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.59817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1165715735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3809959070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.59805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.59811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.59817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1165715735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3809959070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.59805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.59811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.59817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1165715735' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3809959070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:49 smithi196 ceph-mon[76349]: from='client.69791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:49 smithi196 ceph-mon[76349]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:49 smithi184 ceph-mon[100622]: from='client.69791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:49 smithi184 ceph-mon[100622]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:49 smithi184 ceph-mon[102844]: from='client.69791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:49 smithi184 ceph-mon[102844]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:51.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:51 smithi184 ceph-mon[100622]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:51.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:51 smithi184 ceph-mon[102844]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:51.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:51.132Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:51.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:51.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:44:51.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:44:51.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:44:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:51 smithi196 ceph-mon[76349]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:44:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:44:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[100622]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[102844]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:53 smithi196 ceph-mon[76349]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:53.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:53.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:44:55.565 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:55 smithi184 ceph-mon[100622]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:55.565 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:55 smithi184 ceph-mon[102844]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:55 smithi196 ceph-mon[76349]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:57 smithi184 ceph-mon[100622]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:57 smithi184 ceph-mon[102844]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:57 smithi196 ceph-mon[76349]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:59 smithi184 ceph-mon[100622]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:44:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:59 smithi184 ceph-mon[102844]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:44:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:59 smithi196 ceph-mon[76349]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:44:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:01.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:01.132Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:01.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:01.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:01.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:01.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:01 smithi184 ceph-mon[100622]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:01 smithi184 ceph-mon[102844]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:01 smithi196 ceph-mon[76349]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:45:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:03 smithi196 ceph-mon[76349]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:03 smithi184 ceph-mon[100622]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:03 smithi184 ceph-mon[102844]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:04 smithi184 ceph-mon[100622]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:04 smithi184 ceph-mon[102844]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:04 smithi196 ceph-mon[76349]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:07 smithi184 ceph-mon[100622]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:07 smithi184 ceph-mon[102844]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:07 smithi196 ceph-mon[76349]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:09 smithi184 ceph-mon[100622]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:09 smithi184 ceph-mon[102844]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:09 smithi196 ceph-mon[76349]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:11 smithi184 ceph-mon[100622]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:11 smithi184 ceph-mon[102844]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:11.133Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:11.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:11.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:11 smithi196 ceph-mon[76349]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:45:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[100622]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:12 smithi184 ceph-mon[102844]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:12 smithi196 ceph-mon[76349]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:14.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:14.394 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:15 smithi184 ceph-mon[100622]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:15 smithi184 ceph-mon[102844]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:15 smithi196 ceph-mon[76349]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:17.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:17 smithi184 ceph-mon[100622]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:17 smithi184 ceph-mon[102844]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:17 smithi196 ceph-mon[76349]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:18.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:18.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:18.546 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:45:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:19.071 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 4h - - 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 7s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 5s ago 3h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 7s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 7s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 7s ago 4h 404M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 5s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 7s ago 4h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 7s ago 3h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 5s ago 3h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 7s ago 4h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:45:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 7s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 3h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 5s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 5s ago 3h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 5s ago 3h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 5s ago 3h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 5s ago 3h 69.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:45:19.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 5s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:45:19.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:45:19.385 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:45:19.385 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:19 smithi184 ceph-mon[100622]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:19.386 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:19 smithi184 ceph-mon[102844]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:19 smithi196 ceph-mon[76349]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:45:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:45:19.658 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:45:19.658 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:45:19.658 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:45:19.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:45:19.966 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:45:19.966 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:19.966 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:19.966 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:19.966 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[102844]: from='client.69803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[102844]: from='client.59847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[102844]: from='client.59853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2101682272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1035552869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[100622]: from='client.69803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[100622]: from='client.59847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[100622]: from='client.59853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2101682272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:20.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1035552869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:20 smithi196 ceph-mon[76349]: from='client.69803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:20 smithi196 ceph-mon[76349]: from='client.59847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:20 smithi196 ceph-mon[76349]: from='client.59853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2101682272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1035552869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-mon[102844]: from='client.59865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-mon[102844]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-mon[100622]: from='client.59865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-mon[100622]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:21.134Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:21.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:21.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:21 smithi196 ceph-mon[76349]: from='client.59865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:21 smithi196 ceph-mon[76349]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:45:23.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:23 smithi196 ceph-mon[76349]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:23 smithi184 ceph-mon[102844]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:23 smithi184 ceph-mon[100622]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:25.516 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:25 smithi196 ceph-mon[76349]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T23:45:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:25 smithi184 ceph-mon[100622]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T23:45:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:25 smithi184 ceph-mon[102844]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-20T23:45:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:27 smithi184 ceph-mon[100622]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:27 smithi184 ceph-mon[102844]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:27 smithi196 ceph-mon[76349]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:29 smithi184 ceph-mon[100622]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:29 smithi184 ceph-mon[102844]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:29 smithi196 ceph-mon[76349]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:45:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:31 smithi184 ceph-mon[100622]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:31 smithi184 ceph-mon[102844]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:31.135Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:31.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:31.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:31 smithi196 ceph-mon[76349]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:32.286 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[100622]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:32 smithi184 ceph-mon[102844]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:32 smithi196 ceph-mon[76349]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:35.325 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:35 smithi184 ceph-mon[100622]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:35.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:35 smithi184 ceph-mon[102844]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:35 smithi196 ceph-mon[76349]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:45:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:37 smithi184 ceph-mon[100622]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:37 smithi184 ceph-mon[102844]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:37 smithi196 ceph-mon[76349]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:39 smithi184 ceph-mon[100622]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:39 smithi184 ceph-mon[102844]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:39 smithi196 ceph-mon[76349]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:41 smithi184 ceph-mon[100622]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:41 smithi184 ceph-mon[102844]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:41.136Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:41.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:41.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:41 smithi196 ceph-mon[76349]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:45:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:43 smithi184 ceph-mon[100622]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:43 smithi184 ceph-mon[102844]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:43 smithi196 ceph-mon[76349]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:45:44.370 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:45 smithi196 ceph-mon[76349]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:45 smithi184 ceph-mon[100622]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:45 smithi184 ceph-mon[102844]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:45:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:47 smithi184 ceph-mon[100622]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:47 smithi184 ceph-mon[102844]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:47 smithi196 ceph-mon[76349]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:49 smithi184 ceph-mon[100622]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:49 smithi184 ceph-mon[102844]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:49 smithi196 ceph-mon[76349]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:50.262 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 19s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 17s ago 3h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 4h 405M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 17s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:45:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 19s ago 3h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 17s ago 3h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 4h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 3h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 3h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 3h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 3h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 17s ago 3h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:45:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:45:50.791 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:45:51.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:45:51.102 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:45:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:45:51.373 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:45:51.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[100622]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:51.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[100622]: from='client.69839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:51.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/663985117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:51.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[102844]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:51.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[102844]: from='client.69839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:51.374 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/663985117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:51.374 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:51.136Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:51.374 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:51.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:45:51.374 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:45:51.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:51 smithi196 ceph-mon[76349]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:51 smithi196 ceph-mon[76349]: from='client.69839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/663985117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:51.683 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:45:51.684 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:51.684 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:51.684 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:51.684 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:52.294 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:45:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='client.69845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='client.59889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='client.59901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/329255692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='client.69845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='client.59889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='client.59901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/329255692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='client.69845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='client.59889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='client.59901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/329255692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:53 smithi184 ceph-mon[100622]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:53 smithi184 ceph-mon[102844]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:53.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:53 smithi196 ceph-mon[76349]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:45:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:55 smithi196 ceph-mon[76349]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:55 smithi184 ceph-mon[100622]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:55 smithi184 ceph-mon[102844]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:57 smithi196 ceph-mon[76349]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:57.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:57 smithi184 ceph-mon[100622]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:57.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:57 smithi184 ceph-mon[102844]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:59 smithi184 ceph-mon[100622]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:59 smithi184 ceph-mon[102844]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:59 smithi196 ceph-mon[76349]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:01.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:01.137Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:01.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:01.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:01.394 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:01.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:01 smithi184 ceph-mon[100622]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:01 smithi184 ceph-mon[102844]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:01 smithi196 ceph-mon[76349]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:46:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:03 smithi184 ceph-mon[100622]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:03 smithi184 ceph-mon[102844]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:03 smithi196 ceph-mon[76349]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:05 smithi196 ceph-mon[76349]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:05 smithi184 ceph-mon[100622]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:05 smithi184 ceph-mon[102844]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:07 smithi184 ceph-mon[100622]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:07 smithi184 ceph-mon[102844]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:07 smithi196 ceph-mon[76349]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:09 smithi184 ceph-mon[100622]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:09 smithi184 ceph-mon[102844]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:09 smithi196 ceph-mon[76349]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:11.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:11.138Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:11.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:11.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:11.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:11.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:11 smithi184 ceph-mon[100622]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:11 smithi184 ceph-mon[102844]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:11 smithi196 ceph-mon[76349]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[100622]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:12 smithi184 ceph-mon[102844]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:12 smithi196 ceph-mon[76349]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:15 smithi184 ceph-mon[100622]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:15 smithi184 ceph-mon[102844]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:15 smithi196 ceph-mon[76349]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:16 smithi184 ceph-mon[100622]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:16 smithi184 ceph-mon[102844]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:16 smithi196 ceph-mon[76349]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:18.618 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:18 smithi184 ceph-mon[100622]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:18.618 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:18 smithi184 ceph-mon[102844]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:18 smithi196 ceph-mon[76349]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:21 smithi184 ceph-mon[100622]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:21 smithi184 ceph-mon[102844]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:21.139Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:21.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:21.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:21 smithi196 ceph-mon[76349]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:21.980 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:46:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:46:22.511 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 4h - - 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 10s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 9s ago 3h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 10s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 10s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 10s ago 4h 406M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 10s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 10s ago 3h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 9s ago 3h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 10s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:46:22.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 10s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 10s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 10s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 3h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 3h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 3h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 3h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:46:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:22.822 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:46:22.823 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:46:23.091 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:46:23.091 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:46:23.091 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:46:23.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:46:23.401 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:46:23.401 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:23.402 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[100622]: from='client.59913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[100622]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[100622]: from='client.69881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[100622]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/14702840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:23.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[102844]: from='client.59913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[102844]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[102844]: from='client.69881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[102844]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/14702840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:23 smithi196 ceph-mon[76349]: from='client.59913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:23 smithi196 ceph-mon[76349]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:23 smithi196 ceph-mon[76349]: from='client.69881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:23 smithi196 ceph-mon[76349]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/14702840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:24 smithi184 ceph-mon[100622]: from='client.59937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2841370044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:24 smithi184 ceph-mon[102844]: from='client.59937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2841370044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:24 smithi196 ceph-mon[76349]: from='client.59937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2841370044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:25 smithi196 ceph-mon[76349]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:25 smithi184 ceph-mon[100622]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:25 smithi184 ceph-mon[102844]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:27.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:27 smithi196 ceph-mon[76349]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:27 smithi184 ceph-mon[100622]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:27 smithi184 ceph-mon[102844]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:29 smithi196 ceph-mon[76349]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:29 smithi184 ceph-mon[100622]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:29 smithi184 ceph-mon[102844]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:31 smithi184 ceph-mon[100622]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:31 smithi184 ceph-mon[102844]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:31.140Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:31.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:31.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:31 smithi196 ceph-mon[76349]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:46:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:33 smithi196 ceph-mon[76349]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[100622]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:33.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:33 smithi184 ceph-mon[102844]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:34 smithi196 ceph-mon[76349]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[102844]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:35.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:34 smithi184 ceph-mon[100622]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:37.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:37 smithi184 ceph-mon[100622]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:37.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:37 smithi184 ceph-mon[102844]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:37 smithi196 ceph-mon[76349]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:39.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:39 smithi184 ceph-mon[100622]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:39.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:39 smithi184 ceph-mon[102844]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:39 smithi196 ceph-mon[76349]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:41 smithi184 ceph-mon[100622]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:41 smithi184 ceph-mon[102844]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:41.141Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:41.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:41.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:41 smithi196 ceph-mon[76349]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:43 smithi184 ceph-mon[100622]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:43 smithi184 ceph-mon[102844]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:43 smithi196 ceph-mon[76349]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:45 smithi184 ceph-mon[100622]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:45 smithi184 ceph-mon[102844]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:45 smithi196 ceph-mon[76349]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:47 smithi196 ceph-mon[76349]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:47 smithi184 ceph-mon[100622]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:47 smithi184 ceph-mon[102844]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:49 smithi184 ceph-mon[100622]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:49 smithi184 ceph-mon[102844]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:49 smithi196 ceph-mon[76349]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:51 smithi184 ceph-mon[100622]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:51 smithi184 ceph-mon[102844]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:51.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:51.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:46:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:46:51.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:46:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:51 smithi196 ceph-mon[76349]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:46:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:46:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[100622]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[102844]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:53 smithi196 ceph-mon[76349]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:53.685 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 1s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 0s ago 3h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 1s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 1s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 1s ago 4h 406M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 0s ago 4h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:46:54.219 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 1s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 1s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 0s ago 3h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 1s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 1s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 1s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 4h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 0s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 0s ago 3h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 0s ago 3h 73.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 0s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:46:54.220 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:46:54.528 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:46:54.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:46:54.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:46:54.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:46:54.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:46:54.530 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:46:54.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:46:54.799 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:46:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='client.59949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='client.69917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='client.59961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3056079665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='client.59949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='client.69917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='client.59961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:54.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3056079665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:55.107 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:46:55.107 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:55.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:55.107 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:55.108 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='client.59949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='client.69917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='client.59961 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3056079665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:55 smithi196 ceph-mon[76349]: from='client.69935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3654096020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:55 smithi184 ceph-mon[100622]: from='client.69935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3654096020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:55 smithi184 ceph-mon[102844]: from='client.69935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3654096020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:56 smithi184 ceph-mon[100622]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:56 smithi184 ceph-mon[102844]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:56 smithi196 ceph-mon[76349]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:58.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:58.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:58.972 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:58 smithi184 ceph-mon[100622]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:58.973 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:58 smithi184 ceph-mon[102844]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:58 smithi196 ceph-mon[76349]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:00 smithi184 ceph-mon[100622]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:00 smithi184 ceph-mon[102844]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:00 smithi196 ceph-mon[76349]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:01.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:01.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:01.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:02] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T23:47:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:03 smithi184 ceph-mon[100622]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:03 smithi184 ceph-mon[102844]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:03 smithi196 ceph-mon[76349]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:05 smithi196 ceph-mon[76349]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:05 smithi184 ceph-mon[100622]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:05 smithi184 ceph-mon[102844]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:07.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:07 smithi196 ceph-mon[76349]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:07 smithi184 ceph-mon[100622]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:07 smithi184 ceph-mon[102844]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:08.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:09 smithi184 ceph-mon[100622]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:09 smithi184 ceph-mon[102844]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:09 smithi196 ceph-mon[76349]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:11.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:11 smithi184 ceph-mon[102844]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:11.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:11 smithi184 ceph-mon[100622]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:11.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:11.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:11.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:11.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:11.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:11.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:11 smithi196 ceph-mon[76349]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:12] "GET /metrics HTTP/1.1" 200 38929 "" "Prometheus/2.43.0" 2024-09-20T23:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:13 smithi196 ceph-mon[76349]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[100622]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[102844]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:14 smithi196 ceph-mon[76349]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[102844]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:14 smithi184 ceph-mon[100622]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:17 smithi184 ceph-mon[100622]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:17 smithi184 ceph-mon[102844]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:17 smithi196 ceph-mon[76349]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:19.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:19 smithi184 ceph-mon[100622]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:19 smithi184 ceph-mon[102844]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:19 smithi196 ceph-mon[76349]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:21 smithi184 ceph-mon[100622]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:21 smithi184 ceph-mon[102844]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:21.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:21.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:21.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:21 smithi196 ceph-mon[76349]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:22] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:47:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:23 smithi184 ceph-mon[100622]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:23 smithi184 ceph-mon[102844]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:23 smithi196 ceph-mon[76349]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:25.408 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:47:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:25 smithi184 ceph-mon[100622]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:25 smithi184 ceph-mon[102844]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:25 smithi196 ceph-mon[76349]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 4h - - 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 13s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 12s ago 3h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 13s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:47:25.935 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 13s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 13s ago 4h 407M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 4h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 13s ago 4h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 13s ago 3h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 12s ago 3h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 13s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 13s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 13s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 13s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 4h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 3h 73.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 13s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:47:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:47:25.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 13s ago 3h 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:47:25.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:47:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:47:26.249 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:47:26.517 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:47:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:47:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:47:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:47:26.518 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:47:26.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:26 smithi184 ceph-mon[100622]: from='client.69947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:26.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:26 smithi184 ceph-mon[102844]: from='client.69947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:26 smithi196 ceph-mon[76349]: from='client.69947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:26.828 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:47:26.829 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:26.829 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:26.829 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:26.829 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:27.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:27 smithi196 ceph-mon[76349]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:27 smithi196 ceph-mon[76349]: from='client.59997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:27 smithi196 ceph-mon[76349]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/876347479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:27.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2863704146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[102844]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[102844]: from='client.59997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[102844]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/876347479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2863704146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[100622]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[100622]: from='client.59997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[100622]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/876347479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:27.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2863704146' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:28 smithi184 ceph-mon[102844]: from='client.69971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:28 smithi184 ceph-mon[100622]: from='client.69971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:28.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:28 smithi196 ceph-mon[76349]: from='client.69971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:29 smithi196 ceph-mon[76349]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:29 smithi184 ceph-mon[100622]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:29 smithi184 ceph-mon[102844]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:31 smithi184 ceph-mon[100622]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:31 smithi184 ceph-mon[102844]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:31.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:31.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:31.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:31 smithi196 ceph-mon[76349]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:32] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[100622]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[102844]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:33 smithi196 ceph-mon[76349]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:33.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[100622]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:35 smithi184 ceph-mon[102844]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:35 smithi196 ceph-mon[76349]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:37 smithi184 ceph-mon[100622]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:37 smithi184 ceph-mon[102844]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:37 smithi196 ceph-mon[76349]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:39 smithi184 ceph-mon[100622]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:39 smithi184 ceph-mon[102844]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:39 smithi196 ceph-mon[76349]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:41 smithi184 ceph-mon[100622]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:41 smithi184 ceph-mon[102844]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:41.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:41.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:41.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:41 smithi196 ceph-mon[76349]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:42] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:47:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:43 smithi184 ceph-mon[100622]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:43 smithi184 ceph-mon[102844]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:43 smithi196 ceph-mon[76349]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:45 smithi184 ceph-mon[102844]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:45 smithi184 ceph-mon[100622]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:45 smithi196 ceph-mon[76349]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:47 smithi184 ceph-mon[102844]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:47 smithi184 ceph-mon[100622]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:47 smithi196 ceph-mon[76349]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:49 smithi184 ceph-mon[102844]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:49 smithi184 ceph-mon[100622]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:49 smithi196 ceph-mon[76349]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:51 smithi184 ceph-mon[100622]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:51 smithi184 ceph-mon[102844]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:51.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:51.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:47:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:47:51.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:47:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:51 smithi196 ceph-mon[76349]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:47:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:47:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[102844]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[100622]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:53 smithi196 ceph-mon[76349]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:55 smithi196 ceph-mon[76349]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[102844]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:47:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:55 smithi184 ceph-mon[100622]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:57.122 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:47:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:57 smithi184 ceph-mon[100622]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:57 smithi184 ceph-mon[102844]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:57 smithi196 ceph-mon[76349]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:57.650 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:57.650 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 4h - - 2024-09-20T23:47:57.650 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 4h - - 2024-09-20T23:47:57.650 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 5s ago 3h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:47:57.650 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (3h) 3s ago 3h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 3s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 4h 408M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 3s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (3h) 5s ago 3h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 3s ago 3h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 3s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:47:57.651 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 3s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 3s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 3s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 3s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:47:57.652 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 3s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:47:57.962 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:47:57.962 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:47:57.963 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:47:57.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:47:57.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:47:57.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:47:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:47:58.235 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:47:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[100622]: from='client.69983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[100622]: from='client.69989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1566318572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[102844]: from='client.69983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[102844]: from='client.69989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1566318572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:58.549 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:47:58.549 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:58.549 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:58.549 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:58.549 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:58 smithi196 ceph-mon[76349]: from='client.69983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:58 smithi196 ceph-mon[76349]: from='client.69989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1566318572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[100622]: from='client.69995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[100622]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[100622]: from='client.70007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/26133666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[102844]: from='client.69995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[102844]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[102844]: from='client.70007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/26133666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:59 smithi196 ceph-mon[76349]: from='client.69995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:59 smithi196 ceph-mon[76349]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:59 smithi196 ceph-mon[76349]: from='client.70007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/26133666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:01.227 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:01.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:01.227 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:01.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:01.227 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:01.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:01 smithi184 ceph-mon[100622]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:01 smithi184 ceph-mon[102844]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:01 smithi196 ceph-mon[76349]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:48:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:03 smithi184 ceph-mon[100622]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:03 smithi184 ceph-mon[102844]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:03 smithi196 ceph-mon[76349]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:05 smithi184 ceph-mon[100622]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:05 smithi184 ceph-mon[102844]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:05 smithi196 ceph-mon[76349]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:07 smithi184 ceph-mon[100622]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:07 smithi184 ceph-mon[102844]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:07 smithi196 ceph-mon[76349]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:08.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:09.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:09 smithi196 ceph-mon[76349]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:09 smithi184 ceph-mon[100622]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:09 smithi184 ceph-mon[102844]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:11 smithi184 ceph-mon[100622]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:11 smithi184 ceph-mon[102844]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:11.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:11.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:11.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:11 smithi196 ceph-mon[76349]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:48:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[100622]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[102844]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:13 smithi196 ceph-mon[76349]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:48:13.813513083Z level=info msg="Completed cleanup jobs" duration=14.029166ms 2024-09-20T23:48:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:15 smithi184 ceph-mon[100622]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:15 smithi184 ceph-mon[102844]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:15 smithi196 ceph-mon[76349]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:17 smithi184 ceph-mon[100622]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:17 smithi184 ceph-mon[102844]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:17 smithi196 ceph-mon[76349]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:19 smithi184 ceph-mon[100622]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:19 smithi184 ceph-mon[102844]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:19 smithi196 ceph-mon[76349]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:21.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:21.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:21.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:21.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:21.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:21.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:21 smithi184 ceph-mon[100622]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:21 smithi184 ceph-mon[102844]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:21 smithi196 ceph-mon[76349]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:48:23.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:23 smithi184 ceph-mon[100622]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:23.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:23 smithi184 ceph-mon[102844]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:23 smithi196 ceph-mon[76349]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:25 smithi196 ceph-mon[76349]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:25 smithi184 ceph-mon[100622]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:25 smithi184 ceph-mon[102844]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:27 smithi196 ceph-mon[76349]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:27 smithi184 ceph-mon[100622]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:27 smithi184 ceph-mon[102844]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 4h - - 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 16s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 16s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:48:29.373 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 15s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 16s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 16s ago 4h 409M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 15s ago 4h 277M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 16s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (3h) 15s ago 3h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 16s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 16s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 16s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 16s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 15s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 15s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 15s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:48:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 15s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:48:29.375 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 15s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:48:29.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:48:29.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 15s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:48:29.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:48:29.375 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 15s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:48:29.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:29 smithi196 ceph-mon[76349]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:29.680 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:48:29.681 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:48:29.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:48:29.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:48:29.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:48:29.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:29 smithi184 ceph-mon[100622]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:29.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:29 smithi184 ceph-mon[102844]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:48:29.950 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:48:29.951 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:48:30.259 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:48:30.260 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:30.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:30.260 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:30.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[100622]: from='client.60057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[100622]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[100622]: from='client.60069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1319254122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/301851007' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[102844]: from='client.60057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[102844]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[102844]: from='client.60069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1319254122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/301851007' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:30 smithi196 ceph-mon[76349]: from='client.60057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:30 smithi196 ceph-mon[76349]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:30 smithi196 ceph-mon[76349]: from='client.60069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1319254122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/301851007' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:31 smithi196 ceph-mon[76349]: from='client.70043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:31 smithi196 ceph-mon[76349]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-mon[102844]: from='client.70043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-mon[102844]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-mon[100622]: from='client.70043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:31.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-mon[100622]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:31.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:31.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:31.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:48:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[100622]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[102844]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:33 smithi196 ceph-mon[76349]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:35 smithi196 ceph-mon[76349]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[102844]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[100622]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:37 smithi196 ceph-mon[76349]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:37 smithi184 ceph-mon[100622]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:37 smithi184 ceph-mon[102844]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:39 smithi184 ceph-mon[100622]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:39 smithi184 ceph-mon[102844]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:39 smithi196 ceph-mon[76349]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:41.401 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:41.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:41.401 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:41.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:41.402 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:41.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:41.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:41 smithi196 ceph-mon[76349]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:41 smithi184 ceph-mon[100622]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:41 smithi184 ceph-mon[102844]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:42.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:48:43.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:43 smithi184 ceph-mon[102844]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:43.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:43 smithi184 ceph-mon[100622]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:43 smithi196 ceph-mon[76349]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:45 smithi184 ceph-mon[100622]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:45 smithi184 ceph-mon[102844]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:45 smithi196 ceph-mon[76349]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:47 smithi184 ceph-mon[100622]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:47 smithi184 ceph-mon[102844]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:47 smithi196 ceph-mon[76349]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:49 smithi196 ceph-mon[76349]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:49 smithi184 ceph-mon[100622]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:49 smithi184 ceph-mon[102844]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:51.428 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:51.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:51.428 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:51.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:48:51.428 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:48:51.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:48:51.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:51 smithi196 ceph-mon[76349]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:51 smithi184 ceph-mon[100622]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:51 smithi184 ceph-mon[102844]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:48:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:48:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[100622]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[102844]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:53 smithi196 ceph-mon[76349]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[100622]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[102844]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:55 smithi196 ceph-mon[76349]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:48:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:57 smithi184 ceph-mon[100622]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:57 smithi184 ceph-mon[102844]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:57 smithi196 ceph-mon[76349]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:59 smithi184 ceph-mon[100622]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:59 smithi184 ceph-mon[102844]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:59 smithi196 ceph-mon[76349]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:00.558 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 4h - - 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (3h) 8s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:49:01.083 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 4h 410M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 6s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:49:01.084 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 6s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 6s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 6s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 6s ago 3h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 8s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:49:01.085 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 6s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:49:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:49:01.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:49:01.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:01.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:01.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:01.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:01.392 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:01.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:01.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:49:01.662 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:49:01.662 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:49:01.662 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:49:01.663 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:49:01.663 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-mon[100622]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:01.663 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2035360548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:01.664 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-mon[102844]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:01.664 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2035360548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:01 smithi196 ceph-mon[76349]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2035360548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:01.988 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:49:01.988 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:01.988 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[100622]: from='client.60093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[100622]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[100622]: from='client.60105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1883425499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[102844]: from='client.60093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[102844]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[102844]: from='client.60105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1883425499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:02 smithi196 ceph-mon[76349]: from='client.60093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:02 smithi196 ceph-mon[76349]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:02 smithi196 ceph-mon[76349]: from='client.60105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1883425499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:03.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:03 smithi184 ceph-mon[100622]: from='client.60117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:03.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:03 smithi184 ceph-mon[100622]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:03.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:03 smithi184 ceph-mon[102844]: from='client.60117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:03.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:03 smithi184 ceph-mon[102844]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:03 smithi196 ceph-mon[76349]: from='client.60117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:03 smithi196 ceph-mon[76349]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:04 smithi196 ceph-mon[76349]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:05.009 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:04 smithi184 ceph-mon[100622]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:05.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:04 smithi184 ceph-mon[102844]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:07 smithi184 ceph-mon[100622]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:07 smithi184 ceph-mon[102844]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:07 smithi196 ceph-mon[76349]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:09 smithi184 ceph-mon[100622]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:09 smithi184 ceph-mon[102844]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:09 smithi196 ceph-mon[76349]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:11.503 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:11 smithi196 ceph-mon[76349]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:11 smithi184 ceph-mon[100622]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:11 smithi184 ceph-mon[102844]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:11.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:11.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:11.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:49:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[102844]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[100622]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:13 smithi196 ceph-mon[76349]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[100622]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[102844]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:15 smithi196 ceph-mon[76349]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:17 smithi184 ceph-mon[100622]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:17 smithi184 ceph-mon[102844]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:17 smithi196 ceph-mon[76349]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:19 smithi184 ceph-mon[100622]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:19 smithi184 ceph-mon[102844]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:19 smithi196 ceph-mon[76349]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:21 smithi184 ceph-mon[100622]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:21 smithi184 ceph-mon[102844]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:21.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:21.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:21.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:21 smithi196 ceph-mon[76349]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:49:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:23 smithi196 ceph-mon[76349]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:23.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:23 smithi184 ceph-mon[100622]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:23.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:23 smithi184 ceph-mon[102844]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:25 smithi196 ceph-mon[76349]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:25 smithi184 ceph-mon[100622]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:25 smithi184 ceph-mon[102844]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:27 smithi196 ceph-mon[76349]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:27 smithi184 ceph-mon[100622]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:27 smithi184 ceph-mon[102844]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:29 smithi196 ceph-mon[76349]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:29 smithi184 ceph-mon[100622]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:29 smithi184 ceph-mon[102844]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:31 smithi184 ceph-mon[100622]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:31.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:31 smithi184 ceph-mon[102844]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:31.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:31.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:31.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:31 smithi196 ceph-mon[76349]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:32.285 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:49:32.285 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 4h - - 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 18s ago 4h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 19s ago 3h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 19s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:49:32.809 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 19s ago 4h 411M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 19s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 18.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 19s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 19s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 19s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 19s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:49:32.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:49:32.811 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:49:32.811 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 3h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:49:32.811 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 3h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:49:33.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:49:33.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:49:33.120 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:49:33.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:49:33.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:49:33.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:49:33.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:33.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: from='client.60129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:33.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/401843938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:33.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: from='client.60129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/401843938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.403 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:49:33.403 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:49:33.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: from='client.60129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/401843938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:33.727 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:49:33.727 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:33.727 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:33.727 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:33.727 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:34 smithi196 ceph-mon[76349]: from='client.70097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:34 smithi196 ceph-mon[76349]: from='client.70103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:34 smithi196 ceph-mon[76349]: from='client.70115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3538659902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[100622]: from='client.70097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[100622]: from='client.70103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[100622]: from='client.70115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3538659902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[102844]: from='client.70097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[102844]: from='client.70103 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[102844]: from='client.70115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3538659902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[100622]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[102844]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:35 smithi196 ceph-mon[76349]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:37 smithi196 ceph-mon[76349]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:37 smithi184 ceph-mon[100622]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:37 smithi184 ceph-mon[102844]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:39 smithi196 ceph-mon[76349]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:39 smithi184 ceph-mon[100622]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:39 smithi184 ceph-mon[102844]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:41 smithi184 ceph-mon[100622]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:41 smithi184 ceph-mon[102844]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:41.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:41.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:41.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:41 smithi196 ceph-mon[76349]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:49:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:43 smithi196 ceph-mon[76349]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:43.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:43 smithi184 ceph-mon[100622]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:43.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:43 smithi184 ceph-mon[102844]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:44.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:44.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:45 smithi184 ceph-mon[100622]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:45 smithi184 ceph-mon[102844]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:45 smithi196 ceph-mon[76349]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:47 smithi196 ceph-mon[76349]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:47 smithi184 ceph-mon[102844]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:47 smithi184 ceph-mon[100622]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:49 smithi196 ceph-mon[76349]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:49 smithi184 ceph-mon[100622]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:49 smithi184 ceph-mon[102844]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:51 smithi184 ceph-mon[100622]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:51 smithi184 ceph-mon[102844]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:51.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:51.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:49:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:49:51.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:49:51.576 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:51 smithi196 ceph-mon[76349]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:49:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:49:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:53 smithi196 ceph-mon[76349]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:53 smithi184 ceph-mon[102844]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:53 smithi184 ceph-mon[100622]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:54.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:55 smithi196 ceph-mon[76349]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[102844]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[100622]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:55.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:49:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:57 smithi184 ceph-mon[100622]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:57 smithi184 ceph-mon[102844]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:57 smithi196 ceph-mon[76349]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:59 smithi184 ceph-mon[100622]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:59 smithi184 ceph-mon[102844]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:59 smithi196 ceph-mon[76349]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:49:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:01.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:01.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:01.410 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:01.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:01.411 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:01.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:01 smithi184 ceph-mon[100622]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:01 smithi184 ceph-mon[102844]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:01 smithi196 ceph-mon[76349]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:50:03.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:03 smithi184 ceph-mon[100622]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:03.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:03 smithi184 ceph-mon[102844]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:03.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:03 smithi196 ceph-mon[76349]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:04.034 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 9s ago 4h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:50:04.568 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 4h 412M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 9s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 9s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 9s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 9s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 9s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 9s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:50:04.569 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 9s ago 4h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:50:04.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:50:04.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:50:04.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:50:04.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:50:04.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:50:04.907 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:50:05.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:50:05.182 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:50:05.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[100622]: from='client.70127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[100622]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:05.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[100622]: from='client.70133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1949076479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:05.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[102844]: from='client.70127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[102844]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:05.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[102844]: from='client.70133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1949076479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:05.516 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:05.516 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:05.516 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:05 smithi196 ceph-mon[76349]: from='client.70127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:05 smithi196 ceph-mon[76349]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:05 smithi196 ceph-mon[76349]: from='client.70133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1949076479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[102844]: from='client.70139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[102844]: from='client.70151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1875161682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[100622]: from='client.70139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[100622]: from='client.70151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1875161682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:06 smithi196 ceph-mon[76349]: from='client.70139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:06 smithi196 ceph-mon[76349]: from='client.70151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1875161682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:07 smithi184 ceph-mon[102844]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:07 smithi184 ceph-mon[100622]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:07 smithi196 ceph-mon[76349]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:09 smithi184 ceph-mon[102844]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:09 smithi184 ceph-mon[100622]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:09 smithi196 ceph-mon[76349]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:11.435 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:11.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:11.435 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:11.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:11.435 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:11.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:11 smithi184 ceph-mon[100622]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:11 smithi184 ceph-mon[102844]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:11 smithi196 ceph-mon[76349]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:50:13.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:13 smithi196 ceph-mon[76349]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:13 smithi184 ceph-mon[100622]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:13 smithi184 ceph-mon[102844]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[100622]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[102844]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:15 smithi196 ceph-mon[76349]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:17 smithi184 ceph-mon[100622]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:17 smithi184 ceph-mon[102844]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:17 smithi196 ceph-mon[76349]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:19 smithi184 ceph-mon[100622]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:19 smithi184 ceph-mon[102844]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:19 smithi196 ceph-mon[76349]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:21.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:21.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:21.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:21.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:21.539 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:21.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:21 smithi184 ceph-mon[100622]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:21 smithi184 ceph-mon[102844]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:21 smithi196 ceph-mon[76349]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:50:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:23 smithi184 ceph-mon[100622]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:23 smithi184 ceph-mon[102844]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:23 smithi196 ceph-mon[76349]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:24.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:24 smithi184 ceph-mon[100622]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:24.660 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:24 smithi184 ceph-mon[102844]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:24 smithi196 ceph-mon[76349]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:27 smithi184 ceph-mon[100622]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:27 smithi184 ceph-mon[102844]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:27 smithi196 ceph-mon[76349]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:29 smithi184 ceph-mon[100622]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:29 smithi184 ceph-mon[102844]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:29 smithi196 ceph-mon[76349]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:31 smithi184 ceph-mon[100622]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:31 smithi184 ceph-mon[102844]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:31.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:31.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:31.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:31 smithi196 ceph-mon[76349]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:32] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:50:33.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:33 smithi196 ceph-mon[76349]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:33.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:33 smithi184 ceph-mon[100622]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:33.656 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:33 smithi184 ceph-mon[102844]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:34.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:34.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:34.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:34.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:34 smithi196 ceph-mon[76349]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[100622]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:34 smithi184 ceph-mon[102844]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:35.810 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:50:36.360 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 4h - - 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 4h - - 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 1s ago 4h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 2s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 2s ago 4h 412M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 2s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:50:36.361 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 1s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 2s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 2s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 2s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 2s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 4h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 2s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 2s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:50:36.362 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:50:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:50:36.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:50:36.673 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:50:36.942 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:50:36.943 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:50:36.943 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:50:36.943 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:50:36.943 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:50:37.255 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:37.256 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:37.256 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:37.256 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:37.256 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:37.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[100622]: from='client.60201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[100622]: from='client.70169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[100622]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:37.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[100622]: from='client.70175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2227730651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:37.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[102844]: from='client.60201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[102844]: from='client.70169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[102844]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:37.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[102844]: from='client.70175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2227730651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:37 smithi196 ceph-mon[76349]: from='client.60201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:37 smithi196 ceph-mon[76349]: from='client.70169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:37 smithi196 ceph-mon[76349]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:37 smithi196 ceph-mon[76349]: from='client.70175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2227730651' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:38 smithi196 ceph-mon[76349]: from='client.70187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/901879834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[102844]: from='client.70187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/901879834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[100622]: from='client.70187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/901879834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:39 smithi196 ceph-mon[76349]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:39 smithi184 ceph-mon[100622]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:39 smithi184 ceph-mon[102844]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:41 smithi184 ceph-mon[100622]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:41 smithi184 ceph-mon[102844]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:41.157Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:41.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:41.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:41 smithi196 ceph-mon[76349]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:42] "GET /metrics HTTP/1.1" 200 38931 "" "Prometheus/2.43.0" 2024-09-20T23:50:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:43 smithi184 ceph-mon[100622]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:43 smithi184 ceph-mon[102844]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:43 smithi196 ceph-mon[76349]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:45 smithi184 ceph-mon[100622]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:45 smithi184 ceph-mon[102844]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:45 smithi196 ceph-mon[76349]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:47.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:47 smithi184 ceph-mon[100622]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:47.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:47 smithi184 ceph-mon[102844]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:47 smithi196 ceph-mon[76349]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:49 smithi196 ceph-mon[76349]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:49 smithi184 ceph-mon[100622]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:49 smithi184 ceph-mon[102844]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:51 smithi184 ceph-mon[100622]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:51 smithi184 ceph-mon[102844]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:51.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:51.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:50:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:50:51.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:50:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:51 smithi196 ceph-mon[76349]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:50:52] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:50:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:53 smithi196 ceph-mon[76349]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:53.790 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:53 smithi184 ceph-mon[100622]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:53.790 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:53 smithi184 ceph-mon[102844]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:54.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:54.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:54.883 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:54.883 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:54 smithi196 ceph-mon[76349]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[100622]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:54 smithi184 ceph-mon[102844]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:50:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:57 smithi196 ceph-mon[76349]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:57 smithi184 ceph-mon[100622]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:57 smithi184 ceph-mon[102844]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:59 smithi184 ceph-mon[100622]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:59 smithi184 ceph-mon[102844]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:59 smithi196 ceph-mon[76349]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:01 smithi184 ceph-mon[100622]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:01 smithi184 ceph-mon[102844]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:01.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:01.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:01.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:01 smithi196 ceph-mon[76349]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:02] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:51:03.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:03 smithi184 ceph-mon[100622]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:03 smithi184 ceph-mon[102844]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:03 smithi196 ceph-mon[76349]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:05 smithi184 ceph-mon[100622]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:05 smithi184 ceph-mon[102844]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:05 smithi196 ceph-mon[76349]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:07 smithi184 ceph-mon[100622]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:07 smithi184 ceph-mon[102844]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:07.549 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:51:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:07 smithi196 ceph-mon[76349]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 14s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 14s ago 4h 413M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 12s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:51:08.088 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 14s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 14s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 14s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 14s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 14s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 12s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 12s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 12s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 12s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 12s ago 4h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 14s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 12s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 14s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:51:08.089 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 12s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:51:08.400 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:51:08.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:51:08.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:51:08.402 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:51:08.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:08.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:08 smithi184 ceph-mon[102844]: from='client.70199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:08.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:08.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:08 smithi184 ceph-mon[100622]: from='client.70199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:08 smithi196 ceph-mon[76349]: from='client.70199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:08.675 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:51:08.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:51:08.984 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:51:08.984 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:08.984 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:08.984 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:08.984 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[100622]: from='client.70205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[100622]: from='client.70211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[100622]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/617211865' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3852174239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[102844]: from='client.70205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[102844]: from='client.70211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[102844]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/617211865' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3852174239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:09 smithi196 ceph-mon[76349]: from='client.70205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:09 smithi196 ceph-mon[76349]: from='client.70211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:09 smithi196 ceph-mon[76349]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/617211865' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3852174239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:10 smithi184 ceph-mon[100622]: from='client.70223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:10 smithi184 ceph-mon[102844]: from='client.70223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:10 smithi196 ceph-mon[76349]: from='client.70223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:11 smithi184 ceph-mon[100622]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:11 smithi184 ceph-mon[102844]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:11.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:11.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:11.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:11 smithi196 ceph-mon[76349]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:12] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:51:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:13 smithi196 ceph-mon[76349]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:13 smithi184 ceph-mon[100622]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:13 smithi184 ceph-mon[102844]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:14.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:14.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:14.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:14.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:14.983 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:14 smithi196 ceph-mon[76349]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[100622]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:14 smithi184 ceph-mon[102844]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:17 smithi184 ceph-mon[100622]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:17 smithi184 ceph-mon[102844]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:17 smithi196 ceph-mon[76349]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:19 smithi184 ceph-mon[100622]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:19 smithi184 ceph-mon[102844]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:19 smithi196 ceph-mon[76349]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:21.415 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:21.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:21.416 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:21.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:21.416 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:21.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:21 smithi184 ceph-mon[100622]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:21 smithi184 ceph-mon[102844]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:21 smithi196 ceph-mon[76349]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:51:23.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:23 smithi196 ceph-mon[76349]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:23 smithi184 ceph-mon[100622]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:23 smithi184 ceph-mon[102844]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:25 smithi184 ceph-mon[100622]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:25 smithi184 ceph-mon[102844]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:25 smithi196 ceph-mon[76349]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:27 smithi184 ceph-mon[100622]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:27 smithi184 ceph-mon[102844]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:27 smithi196 ceph-mon[76349]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:29 smithi184 ceph-mon[100622]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:29 smithi184 ceph-mon[102844]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:29 smithi196 ceph-mon[76349]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:31.434 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:31.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:31.435 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:31.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:31.435 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:31.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:31 smithi184 ceph-mon[102844]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:31 smithi184 ceph-mon[100622]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:31 smithi196 ceph-mon[76349]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:51:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:33 smithi184 ceph-mon[100622]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:33 smithi184 ceph-mon[102844]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:33 smithi196 ceph-mon[76349]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[100622]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:35 smithi184 ceph-mon[102844]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:35.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:35.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:35 smithi196 ceph-mon[76349]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:36.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:37 smithi184 ceph-mon[100622]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:37 smithi184 ceph-mon[102844]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:37 smithi196 ceph-mon[76349]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:39.279 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:51:39.802 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:39.803 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 4h - - 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 5s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 5s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:51:39.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 5s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 5s ago 4h 414M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 5s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 5s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 5s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 5s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 5s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 5s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 4s ago 4h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:51:39.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 5s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:51:39.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:51:39.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 5s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:51:39.808 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:51:39.808 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:39 smithi184 ceph-mon[100622]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:39.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:39 smithi184 ceph-mon[102844]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:39 smithi196 ceph-mon[76349]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:40.113 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:51:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:51:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:51:40.115 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:51:40.382 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:51:40.383 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:51:40.698 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:51:40.698 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:40.698 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:40.698 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:40.699 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[100622]: from='client.70235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[100622]: from='client.70241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3157416814' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[102844]: from='client.70235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[102844]: from='client.70241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3157416814' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:40 smithi196 ceph-mon[76349]: from='client.70235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:40 smithi196 ceph-mon[76349]: from='client.70241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3157416814' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:41.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:41.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:41.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:41 smithi196 ceph-mon[76349]: from='client.60285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:41 smithi196 ceph-mon[76349]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:41 smithi196 ceph-mon[76349]: from='client.70259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2284581111' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[100622]: from='client.60285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[100622]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[100622]: from='client.70259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2284581111' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[102844]: from='client.60285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[102844]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[102844]: from='client.70259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:42.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2284581111' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:51:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:42 smithi196 ceph-mon[76349]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:42.989 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:42 smithi184 ceph-mon[100622]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:42.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:42 smithi184 ceph-mon[102844]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:45 smithi184 ceph-mon[100622]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:45 smithi184 ceph-mon[102844]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:45 smithi196 ceph-mon[76349]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:47.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:47 smithi184 ceph-mon[100622]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:47.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:47 smithi184 ceph-mon[102844]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:47 smithi196 ceph-mon[76349]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:48.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:48.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:49.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:49 smithi184 ceph-mon[100622]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:49.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:49 smithi184 ceph-mon[102844]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:49 smithi196 ceph-mon[76349]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:51.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:51.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:51:51.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:51:51.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:51:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:51 smithi196 ceph-mon[76349]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:51 smithi184 ceph-mon[102844]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:51 smithi184 ceph-mon[100622]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:51:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:51:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:53 smithi196 ceph-mon[76349]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:53 smithi184 ceph-mon[102844]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:53 smithi184 ceph-mon[100622]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:55.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:55 smithi196 ceph-mon[76349]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:55.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[102844]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:55.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:55 smithi184 ceph-mon[100622]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:56 smithi196 ceph-mon[76349]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:56.989 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[100622]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:51:56.990 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:56 smithi184 ceph-mon[102844]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:59 smithi196 ceph-mon[76349]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:59 smithi184 ceph-mon[102844]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:59 smithi184 ceph-mon[100622]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:01 smithi184 ceph-mon[100622]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:01 smithi184 ceph-mon[102844]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:01.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:01.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:01.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:01 smithi196 ceph-mon[76349]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:02] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:52:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:03 smithi196 ceph-mon[76349]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:03 smithi184 ceph-mon[100622]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:03 smithi184 ceph-mon[102844]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:05 smithi196 ceph-mon[76349]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:05 smithi184 ceph-mon[102844]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:05 smithi184 ceph-mon[100622]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:07 smithi196 ceph-mon[76349]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:07.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:07 smithi184 ceph-mon[100622]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:07.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:07 smithi184 ceph-mon[102844]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:09 smithi184 ceph-mon[100622]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:09 smithi184 ceph-mon[102844]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:09 smithi196 ceph-mon[76349]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:10.993 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:52:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:11.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:11.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:11.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 4h - - 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 4h - - 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 17s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 17s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 15s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 17s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 17s ago 4h 413M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:52:11.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 15s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 17s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 17s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 15s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 17s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 17s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 17s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 17s ago 4h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 15s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 15s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 15s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 15s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 15s ago 4h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 17s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:52:11.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 15s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 17s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:52:11.524 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 15s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:52:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:11 smithi196 ceph-mon[76349]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:11 smithi184 ceph-mon[100622]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:11 smithi184 ceph-mon[102844]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:52:11.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:52:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:52:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:52:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:52:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:52:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:52:12.104 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:52:12.104 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:12] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-20T23:52:12.413 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:52:12.413 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:12.413 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:12.413 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:12.413 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[100622]: from='client.70271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[100622]: from='client.70277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[100622]: from='client.70283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/189802570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[102844]: from='client.70271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[102844]: from='client.70277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[102844]: from='client.70283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/189802570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:12 smithi196 ceph-mon[76349]: from='client.70271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:12 smithi196 ceph-mon[76349]: from='client.70277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:12 smithi196 ceph-mon[76349]: from='client.70283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/189802570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:13 smithi196 ceph-mon[76349]: from='client.60333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:13 smithi196 ceph-mon[76349]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3014188734' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[100622]: from='client.60333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[100622]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3014188734' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[102844]: from='client.60333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[102844]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3014188734' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:15 smithi196 ceph-mon[76349]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:15 smithi184 ceph-mon[100622]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:15 smithi184 ceph-mon[102844]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[100622]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:16 smithi184 ceph-mon[102844]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:17.069 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.069 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.069 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:17.070 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:16 smithi196 ceph-mon[76349]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:18.086 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:18 smithi196 ceph-mon[76349]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:18 smithi184 ceph-mon[100622]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:18 smithi184 ceph-mon[102844]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:21 smithi184 ceph-mon[100622]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:21 smithi184 ceph-mon[102844]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:21.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:21.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:21.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:21 smithi196 ceph-mon[76349]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:22] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:52:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:23 smithi196 ceph-mon[76349]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:23 smithi184 ceph-mon[100622]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:23 smithi184 ceph-mon[102844]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:25 smithi196 ceph-mon[76349]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:25 smithi184 ceph-mon[100622]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:25 smithi184 ceph-mon[102844]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:27 smithi196 ceph-mon[76349]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:27 smithi184 ceph-mon[100622]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:27 smithi184 ceph-mon[102844]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:28.602 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:28.602 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:29 smithi184 ceph-mon[100622]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:29.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:29 smithi184 ceph-mon[102844]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:29 smithi196 ceph-mon[76349]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:31.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:31.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:31.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:31.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:31 smithi196 ceph-mon[76349]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:31 smithi184 ceph-mon[100622]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:31 smithi184 ceph-mon[102844]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:52:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:33 smithi196 ceph-mon[76349]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:33 smithi184 ceph-mon[100622]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:33.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:33 smithi184 ceph-mon[102844]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[100622]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[102844]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:35 smithi196 ceph-mon[76349]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[102844]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:36 smithi184 ceph-mon[100622]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:37.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:37.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:36 smithi196 ceph-mon[76349]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:39 smithi196 ceph-mon[76349]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:39 smithi184 ceph-mon[102844]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:39 smithi184 ceph-mon[100622]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:41 smithi184 ceph-mon[102844]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:41 smithi184 ceph-mon[100622]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:41.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:41.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:41.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:41 smithi196 ceph-mon[76349]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:52:42.707 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:52:43.234 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 4h - - 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 4h - - 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 8s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 7s ago 4h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 8s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 7s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 8s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 8s ago 4h 414M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 7s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 8s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 7s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:52:43.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 8s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 8s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 8s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 8s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 7s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 7s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 7s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 7s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 7s ago 4h 74.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 8s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 7s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 8s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:52:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 7s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:52:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:43 smithi184 ceph-mon[100622]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:43 smithi184 ceph-mon[102844]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:43.554 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:52:43.555 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:52:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:43 smithi196 ceph-mon[76349]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:52:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:52:43.824 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:52:44.136 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:52:44.137 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:44.137 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:44.137 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:44.137 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='client.70307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='client.70319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3570219949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3185736848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='client.70307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='client.70319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3570219949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3185736848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='client.70307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='client.70319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3570219949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3185736848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:45 smithi196 ceph-mon[76349]: from='client.60369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:45 smithi196 ceph-mon[76349]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:45 smithi184 ceph-mon[100622]: from='client.60369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:45 smithi184 ceph-mon[100622]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:45 smithi184 ceph-mon[102844]: from='client.60369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:45 smithi184 ceph-mon[102844]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:47 smithi196 ceph-mon[76349]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:52:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:47 smithi184 ceph-mon[100622]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:52:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:47 smithi184 ceph-mon[102844]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:52:49.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:49.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:49 smithi184 ceph-mon[100622]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:49.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:49.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:49 smithi184 ceph-mon[102844]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:49 smithi196 ceph-mon[76349]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:51.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:51.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:51.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:51.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:52:51.268 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:52:51.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:52:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:51 smithi184 ceph-mon[100622]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:51 smithi184 ceph-mon[102844]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:51 smithi196 ceph-mon[76349]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:52:52] "GET /metrics HTTP/1.1" 200 38932 "" "Prometheus/2.43.0" 2024-09-20T23:52:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:53 smithi196 ceph-mon[76349]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:53 smithi184 ceph-mon[100622]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:53 smithi184 ceph-mon[102844]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[100622]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[102844]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:55 smithi196 ceph-mon[76349]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:57 smithi196 ceph-mon[76349]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[100622]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:52:57.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:57 smithi184 ceph-mon[102844]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:59 smithi196 ceph-mon[76349]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[100622]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[102844]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:01 smithi184 ceph-mon[100622]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:01 smithi184 ceph-mon[102844]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:01.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:01.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:01.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:01 smithi196 ceph-mon[76349]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:53:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:03 smithi196 ceph-mon[76349]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:03.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:03 smithi184 ceph-mon[100622]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:03.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:03 smithi184 ceph-mon[102844]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:05 smithi196 ceph-mon[76349]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:05 smithi184 ceph-mon[100622]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:05 smithi184 ceph-mon[102844]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:07 smithi196 ceph-mon[76349]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:07 smithi184 ceph-mon[102844]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:07 smithi184 ceph-mon[100622]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:09 smithi196 ceph-mon[76349]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:09 smithi184 ceph-mon[100622]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:09 smithi184 ceph-mon[102844]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:11 smithi184 ceph-mon[100622]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:11 smithi184 ceph-mon[102844]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:11.168Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:11.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:11.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:11 smithi196 ceph-mon[76349]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:53:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:13 smithi196 ceph-mon[76349]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:13 smithi184 ceph-mon[100622]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:13 smithi184 ceph-mon[102844]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:14.439 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:53:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:14.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:14.672 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:14.970 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 4h - - 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 18s ago 4h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 18s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 0s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 0s ago 4h 417M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 18s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 0s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 0s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 0s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:53:14.971 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 0s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 0s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 18s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 18s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 18s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 18s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 18s ago 4h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 0s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 18s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 0s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:53:14.972 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 18s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:15.280 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:53:15.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:53:15.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:53:15.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: from='client.70343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/98886670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: from='client.70343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/98886670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:53:15.550 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:53:15.551 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:53:15.551 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:53:15.551 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: from='client.70343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/98886670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:15.860 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:53:15.860 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:15.860 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:15.860 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:15.860 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='client.70349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='client.70367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1148803911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='client.70349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='client.70367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1148803911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='client.70349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='client.70367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1148803911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:17 smithi196 ceph-mon[76349]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:17 smithi184 ceph-mon[100622]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:17 smithi184 ceph-mon[102844]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:19 smithi196 ceph-mon[76349]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:19 smithi184 ceph-mon[100622]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:19 smithi184 ceph-mon[102844]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:21 smithi184 ceph-mon[100622]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:21 smithi184 ceph-mon[102844]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:21.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:21.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:21.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:21 smithi196 ceph-mon[76349]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:53:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:23 smithi196 ceph-mon[76349]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:23 smithi184 ceph-mon[100622]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:23 smithi184 ceph-mon[102844]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:25 smithi184 ceph-mon[100622]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:25 smithi184 ceph-mon[102844]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:25 smithi196 ceph-mon[76349]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:27 smithi184 ceph-mon[100622]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:27 smithi184 ceph-mon[102844]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:27 smithi196 ceph-mon[76349]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[100622]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[102844]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:29 smithi196 ceph-mon[76349]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:31.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:31 smithi184 ceph-mon[100622]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:31.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:31 smithi184 ceph-mon[102844]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:31.495 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:31.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:31.496 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:31.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:31.496 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:31.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:31 smithi196 ceph-mon[76349]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:53:33.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:33 smithi196 ceph-mon[76349]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:33 smithi184 ceph-mon[100622]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:33 smithi184 ceph-mon[102844]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[100622]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[102844]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:35 smithi196 ceph-mon[76349]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[100622]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:37.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[102844]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:37.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:37 smithi196 ceph-mon[76349]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:39.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:39.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:39 smithi196 ceph-mon[76349]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:39 smithi184 ceph-mon[100622]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:39 smithi184 ceph-mon[102844]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:41 smithi184 ceph-mon[100622]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:41 smithi184 ceph-mon[102844]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:41.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:41.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:41.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:41 smithi196 ceph-mon[76349]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:53:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:43 smithi196 ceph-mon[76349]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:43 smithi184 ceph-mon[100622]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:43 smithi184 ceph-mon[102844]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:45 smithi196 ceph-mon[76349]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:45 smithi184 ceph-mon[100622]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:45 smithi184 ceph-mon[102844]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:46.157 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:53:46.689 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 93.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:53:46.690 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 11s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 11s ago 4h 417M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 10s ago 4h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 11s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 11s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 11s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:53:46.691 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 11s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 11s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 10s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 10s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 10s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 10s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 10s ago 4h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 10s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:53:46.692 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:53:46.693 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 10s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:53:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:53:47.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:53:47.268 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:53:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[100622]: from='client.70379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[100622]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[100622]: from='client.60423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2998959029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[102844]: from='client.70379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[102844]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[102844]: from='client.60423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2998959029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:47.581 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:53:47.581 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:47.581 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:47.582 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:47.582 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:47 smithi196 ceph-mon[76349]: from='client.70379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:47 smithi196 ceph-mon[76349]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:47 smithi196 ceph-mon[76349]: from='client.60423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2998959029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:48 smithi196 ceph-mon[76349]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:48 smithi196 ceph-mon[76349]: from='client.70403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3030842501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[100622]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[100622]: from='client.70403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3030842501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[102844]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[102844]: from='client.70403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3030842501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:49 smithi196 ceph-mon[76349]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:49 smithi184 ceph-mon[100622]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:49 smithi184 ceph-mon[102844]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:51.449 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:51 smithi184 ceph-mon[100622]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:51.450 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:51 smithi184 ceph-mon[102844]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:51.450 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:51.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:51.450 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:51.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:53:51.450 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:53:51.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:53:51.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:51 smithi196 ceph-mon[76349]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:52.295 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:53:52] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-20T23:53:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:53 smithi196 ceph-mon[76349]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:53.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:53 smithi184 ceph-mon[102844]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:53.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:53 smithi184 ceph-mon[100622]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:55 smithi196 ceph-mon[76349]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[100622]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[102844]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[102844]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[100622]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:57 smithi196 ceph-mon[76349]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:53:59.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:59.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:59 smithi196 ceph-mon[76349]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:59.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[100622]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[102844]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:01.441 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:01.173Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:01.442 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:01.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:01.442 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:01.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:01.780 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:01 smithi196 ceph-mon[76349]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:01 smithi184 ceph-mon[102844]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:01 smithi184 ceph-mon[100622]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:54:03.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:03 smithi184 ceph-mon[100622]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:03.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:03 smithi184 ceph-mon[102844]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:03 smithi196 ceph-mon[76349]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:05 smithi184 ceph-mon[100622]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:05 smithi184 ceph-mon[102844]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:05 smithi196 ceph-mon[76349]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:07 smithi184 ceph-mon[100622]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:54:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:07 smithi184 ceph-mon[102844]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:54:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:07 smithi196 ceph-mon[76349]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T23:54:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:09 smithi184 ceph-mon[100622]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:54:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:09 smithi184 ceph-mon[102844]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:54:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:09 smithi196 ceph-mon[76349]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-20T23:54:11.457 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:11.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:11.457 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:11.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:11.458 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:11.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:11 smithi184 ceph-mon[100622]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:54:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:11 smithi184 ceph-mon[102844]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:54:11.818 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:11 smithi196 ceph-mon[76349]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:54:12.189 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:54:13.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:13 smithi184 ceph-mon[100622]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:54:13.724 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:13 smithi184 ceph-mon[102844]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:54:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:13 smithi196 ceph-mon[76349]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-20T23:54:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[100622]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:54:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[102844]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:54:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:15 smithi196 ceph-mon[76349]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T23:54:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[100622]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[102844]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.886 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:54:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:17 smithi196 ceph-mon[76349]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-20T23:54:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 4h - - 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 1s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 1s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:54:18.440 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (3h) 3s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 3s ago 4h 418M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 1s ago 4h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 3s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 1s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 3s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 3s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 3s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 3s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 1s ago 4h 91.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 1s ago 4h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 1s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 1s ago 4h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 1s ago 4h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:54:18.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 3s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:54:18.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 1s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:54:18.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 3s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:54:18.442 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 1s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:54:18.789 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:18.790 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:54:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:54:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:54:18.791 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:54:19.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:54:19.073 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:54:19.074 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:54:19.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:54:19.380 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:54:19.380 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:19.380 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:19.380 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:19.380 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:19.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:19.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.60453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.70421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:54:19.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3955194733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:19.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1048436333' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.60453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.70421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:54:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3955194733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1048436333' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.60453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.70421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3955194733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1048436333' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:20.780 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:20 smithi196 ceph-mon[76349]: from='client.70439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:20 smithi184 ceph-mon[102844]: from='client.70439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:20 smithi184 ceph-mon[100622]: from='client.70439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:21.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:21.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:21.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:21 smithi196 ceph-mon[76349]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:54:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:21 smithi184 ceph-mon[100622]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:54:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:21 smithi184 ceph-mon[102844]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T23:54:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:54:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:22 smithi196 ceph-mon[76349]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:54:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:22 smithi184 ceph-mon[100622]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:54:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:22 smithi184 ceph-mon[102844]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T23:54:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:25 smithi196 ceph-mon[76349]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:54:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:25 smithi184 ceph-mon[100622]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:54:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:25 smithi184 ceph-mon[102844]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:54:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:27 smithi196 ceph-mon[76349]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:54:27.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:27 smithi184 ceph-mon[100622]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:54:27.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:27 smithi184 ceph-mon[102844]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:54:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[100622]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:54:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[102844]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:54:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:29 smithi196 ceph-mon[76349]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T23:54:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:31 smithi184 ceph-mon[100622]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:54:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:31 smithi184 ceph-mon[102844]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:54:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:31.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:31.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:31.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:31 smithi196 ceph-mon[76349]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T23:54:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:32] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:54:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:33 smithi196 ceph-mon[76349]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:54:33.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:33 smithi184 ceph-mon[100622]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:54:33.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:33 smithi184 ceph-mon[102844]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-20T23:54:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:35 smithi196 ceph-mon[76349]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:54:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[102844]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:54:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[100622]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:37 smithi196 ceph-mon[76349]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:54:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[100622]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:54:37.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[102844]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-20T23:54:37.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:37.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:39 smithi196 ceph-mon[76349]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T23:54:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:39 smithi184 ceph-mon[100622]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T23:54:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:39 smithi184 ceph-mon[102844]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T23:54:41.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:41 smithi196 ceph-mon[76349]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:54:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:41 smithi184 ceph-mon[100622]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:54:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:41 smithi184 ceph-mon[102844]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:54:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:41.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:41.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:41.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:42] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:54:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:43 smithi196 ceph-mon[76349]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:54:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:43 smithi184 ceph-mon[100622]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:54:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:43 smithi184 ceph-mon[102844]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:54:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:45 smithi184 ceph-mon[102844]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T23:54:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:45 smithi184 ceph-mon[100622]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T23:54:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:45 smithi196 ceph-mon[76349]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-20T23:54:47.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:47 smithi184 ceph-mon[102844]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T23:54:47.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:47 smithi184 ceph-mon[100622]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T23:54:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:47 smithi196 ceph-mon[76349]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T23:54:49.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:49.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:49 smithi184 ceph-mon[100622]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:49.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:49.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:49 smithi184 ceph-mon[102844]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:49.679 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:54:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:49 smithi196 ceph-mon[76349]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 4h - - 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 4h - - 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 13s ago 4h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (3h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (3h) 13s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:54:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 15s ago 4h 418M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 13s ago 4h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 15s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 18.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 13s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 15s ago 4h 82.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 15s ago 4h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 15s ago 4h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 15s ago 4h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 13s ago 4h 90.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 13s ago 4h 80.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 13s ago 4h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 13s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:54:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (3h) 13s ago 4h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:54:50.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 15s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:54:50.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 13s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:54:50.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 15s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:54:50.211 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 13s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:54:50.517 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:54:50.517 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:54:50.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:54:50.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:54:50.518 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:54:50.788 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:54:50.789 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:54:50.789 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:54:51.097 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:54:51.097 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:51.097 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:51.097 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:51.098 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:51.408 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:51.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:51.408 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:51.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:54:51.408 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:54:51.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:54:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: from='client.70451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: from='client.70463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/519943847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/731291136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: from='client.70451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: from='client.70463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/519943847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/731291136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: from='client.70451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: from='client.70463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/519943847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/731291136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:54:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:54:52.694 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:52 smithi196 ceph-mon[76349]: from='client.60513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:52 smithi184 ceph-mon[100622]: from='client.60513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:52 smithi184 ceph-mon[102844]: from='client.60513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:53 smithi184 ceph-mon[100622]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:53 smithi184 ceph-mon[102844]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:53 smithi196 ceph-mon[76349]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[102844]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[100622]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:55 smithi196 ceph-mon[76349]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[100622]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[102844]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:57 smithi196 ceph-mon[76349]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:54:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[100622]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[102844]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:54:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:59 smithi196 ceph-mon[76349]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:54:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:01.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:01.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:01.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:01.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:01.432 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:01.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:01.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:01 smithi196 ceph-mon[76349]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:01 smithi184 ceph-mon[100622]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:01 smithi184 ceph-mon[102844]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:55:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:03 smithi196 ceph-mon[76349]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:03 smithi184 ceph-mon[100622]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:03 smithi184 ceph-mon[102844]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:05 smithi184 ceph-mon[102844]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:05 smithi184 ceph-mon[100622]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:05 smithi196 ceph-mon[76349]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:07.692 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:07 smithi184 ceph-mon[100622]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:07.692 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:07 smithi184 ceph-mon[102844]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:07 smithi196 ceph-mon[76349]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:09 smithi184 ceph-mon[102844]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:09 smithi184 ceph-mon[100622]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:09 smithi196 ceph-mon[76349]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:11.446 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:11.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:11.446 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:11.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:11.446 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:11.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:11 smithi184 ceph-mon[100622]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:11 smithi184 ceph-mon[102844]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:11 smithi196 ceph-mon[76349]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:55:13.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:13 smithi184 ceph-mon[100622]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:13.775 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:13 smithi184 ceph-mon[102844]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:13 smithi196 ceph-mon[76349]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[100622]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[102844]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:15 smithi196 ceph-mon[76349]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[102844]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[100622]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:17 smithi196 ceph-mon[76349]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:19 smithi184 ceph-mon[100622]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:19 smithi184 ceph-mon[102844]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:19 smithi196 ceph-mon[76349]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:21.391 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:55:21.471 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:21.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:21.471 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:21.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:21.471 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:21.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:21.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:21 smithi196 ceph-mon[76349]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:21 smithi184 ceph-mon[100622]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:21 smithi184 ceph-mon[102844]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:21.917 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:55:21.917 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 6s ago 4h 419M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 4s ago 4h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 6s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 6s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 6s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 6s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:55:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 6s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 4s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 4s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 4s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 4s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 4h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 6s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 4s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 6s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:55:21.919 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 4s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:55:22.226 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:55:22.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:55:22.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:55:22.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:55:22.228 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:22] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:55:22.497 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:55:22.498 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:55:22.498 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:55:22.498 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:55:22.498 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:55:22.498 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:55:22.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:22 smithi196 ceph-mon[76349]: from='client.60525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:22.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1459421887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:22 smithi184 ceph-mon[100622]: from='client.60525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:22.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1459421887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:22 smithi184 ceph-mon[102844]: from='client.60525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:22.807 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1459421887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:22.807 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:55:22.807 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:22.807 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:22.807 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:22.807 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:23.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:23 smithi196 ceph-mon[76349]: from='client.60531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:23 smithi196 ceph-mon[76349]: from='client.70499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:23 smithi196 ceph-mon[76349]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:23.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:23 smithi196 ceph-mon[76349]: from='client.60549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2845932757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[100622]: from='client.60531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[100622]: from='client.70499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[100622]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[100622]: from='client.60549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2845932757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[102844]: from='client.60531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[102844]: from='client.70499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[102844]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[102844]: from='client.60549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2845932757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:25.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:25 smithi196 ceph-mon[76349]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:25 smithi184 ceph-mon[100622]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:25 smithi184 ceph-mon[102844]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:27.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:27 smithi184 ceph-mon[100622]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:27.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:27 smithi184 ceph-mon[102844]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:27 smithi196 ceph-mon[76349]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[100622]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[102844]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:29 smithi196 ceph-mon[76349]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:31.494 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:31.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:31.494 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:31.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:31.494 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:31.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:31 smithi184 ceph-mon[102844]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:31 smithi184 ceph-mon[100622]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:31 smithi196 ceph-mon[76349]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:55:33.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:33 smithi184 ceph-mon[100622]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:33.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:33 smithi184 ceph-mon[102844]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:33 smithi196 ceph-mon[76349]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:35.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:35 smithi184 ceph-mon[100622]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:35.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:35 smithi184 ceph-mon[102844]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:35.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:35 smithi196 ceph-mon[76349]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:36.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.691 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[100622]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[102844]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:37 smithi196 ceph-mon[76349]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:39 smithi184 ceph-mon[100622]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:39 smithi184 ceph-mon[102844]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:39 smithi196 ceph-mon[76349]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:41.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:41.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:41.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:41 smithi196 ceph-mon[76349]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:42.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:41 smithi184 ceph-mon[100622]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:41 smithi184 ceph-mon[102844]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:55:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:43 smithi196 ceph-mon[76349]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:43.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:43 smithi184 ceph-mon[100622]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:43.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:43 smithi184 ceph-mon[102844]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:45 smithi196 ceph-mon[76349]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:45 smithi184 ceph-mon[100622]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:45 smithi184 ceph-mon[102844]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:47 smithi196 ceph-mon[76349]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:47 smithi184 ceph-mon[100622]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:47 smithi184 ceph-mon[102844]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:49 smithi196 ceph-mon[76349]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:49 smithi184 ceph-mon[100622]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:49 smithi184 ceph-mon[102844]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:51.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:51.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:55:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:55:51.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:55:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:51 smithi196 ceph-mon[76349]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:51 smithi184 ceph-mon[102844]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:51 smithi184 ceph-mon[100622]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:55:52] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:55:53.103 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:55:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (3h) 18s ago 4h 419M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 16s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (3h) 18s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 18s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 18s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 18s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 16s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 16s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 16s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 16s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 70.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:55:53.631 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:55:53.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 16s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:55:53.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:55:53.632 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 16s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:55:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:53 smithi196 ceph-mon[76349]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:53.942 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:55:53.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:55:53.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:55:53.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:55:53.943 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:55:53.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:53 smithi184 ceph-mon[100622]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:53.944 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:53 smithi184 ceph-mon[102844]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:54.213 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:55:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:55:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:55:54.214 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:55:54.524 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:55:54.524 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:54.524 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:54.524 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:54.524 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[100622]: from='client.60561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[100622]: from='client.70529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2006940407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/551103690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[102844]: from='client.60561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[102844]: from='client.70529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2006940407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/551103690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:54 smithi196 ceph-mon[76349]: from='client.60561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:54 smithi196 ceph-mon[76349]: from='client.70529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2006940407' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/551103690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:55.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[100622]: from='client.70535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[100622]: from='client.70547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[100622]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[102844]: from='client.70535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[102844]: from='client.70547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:55 smithi184 ceph-mon[102844]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:55 smithi196 ceph-mon[76349]: from='client.70535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:55 smithi196 ceph-mon[76349]: from='client.70547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:55 smithi196 ceph-mon[76349]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[102844]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:56.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:56 smithi184 ceph-mon[100622]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:57.036 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:57.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:57.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:57.037 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:56 smithi196 ceph-mon[76349]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[100622]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[102844]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:59 smithi196 ceph-mon[76349]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:01 smithi184 ceph-mon[102844]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:01 smithi184 ceph-mon[100622]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:01.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:01.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:01.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:01 smithi196 ceph-mon[76349]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:56:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:03 smithi184 ceph-mon[100622]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:03 smithi184 ceph-mon[102844]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:03.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:03 smithi196 ceph-mon[76349]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:05 smithi196 ceph-mon[76349]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:05 smithi184 ceph-mon[102844]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:05 smithi184 ceph-mon[100622]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:07.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:07 smithi184 ceph-mon[100622]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:07.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:07 smithi184 ceph-mon[102844]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:07 smithi196 ceph-mon[76349]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:09 smithi184 ceph-mon[102844]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:09 smithi184 ceph-mon[100622]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:09 smithi196 ceph-mon[76349]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:11 smithi184 ceph-mon[100622]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:11 smithi184 ceph-mon[102844]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:11.183Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:11.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:11.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:11 smithi196 ceph-mon[76349]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:56:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:13 smithi184 ceph-mon[100622]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:13 smithi184 ceph-mon[102844]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:13 smithi196 ceph-mon[76349]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:15.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:15 smithi184 ceph-mon[100622]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:15.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:15 smithi184 ceph-mon[102844]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:15.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:15 smithi196 ceph-mon[76349]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:17.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[102844]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:16 smithi184 ceph-mon[100622]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:16 smithi196 ceph-mon[76349]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:19 smithi196 ceph-mon[76349]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:19 smithi184 ceph-mon[102844]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:19 smithi184 ceph-mon[100622]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:21 smithi184 ceph-mon[102844]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:21 smithi184 ceph-mon[100622]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:21.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:21.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:21.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:21 smithi196 ceph-mon[76349]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:56:23.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:23 smithi196 ceph-mon[76349]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:23 smithi184 ceph-mon[100622]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:23 smithi184 ceph-mon[102844]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:24.820 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:56:25.346 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:56:25.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 4h - - 2024-09-20T23:56:25.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 4h - - 2024-09-20T23:56:25.346 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 9s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:56:25.346 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 9s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 9s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 4h 421M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (3h) 8s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 9s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 9s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 9s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 9s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 9s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:56:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 8s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 8s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 8s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 9s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 8s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 9s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:56:25.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 8s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:56:25.657 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:56:25.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:56:25.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:25 smithi184 ceph-mon[100622]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:25.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:25 smithi184 ceph-mon[102844]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:25.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:25 smithi196 ceph-mon[76349]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:56:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:56:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:56:25.934 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:56:26.245 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:56:26.245 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:26.245 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[100622]: from='client.70559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[100622]: from='client.60603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[100622]: from='client.60609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1473596384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/521785546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[102844]: from='client.70559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[102844]: from='client.60603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[102844]: from='client.60609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1473596384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/521785546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:26 smithi196 ceph-mon[76349]: from='client.70559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:26 smithi196 ceph-mon[76349]: from='client.60603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:26 smithi196 ceph-mon[76349]: from='client.60609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1473596384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/521785546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:27.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:27 smithi184 ceph-mon[102844]: from='client.70583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:27.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:27 smithi184 ceph-mon[102844]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:27.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:27 smithi184 ceph-mon[100622]: from='client.70583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:27.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:27 smithi184 ceph-mon[100622]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:27 smithi196 ceph-mon[76349]: from='client.70583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:27 smithi196 ceph-mon[76349]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[100622]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[102844]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:29 smithi196 ceph-mon[76349]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:31 smithi184 ceph-mon[100622]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:31 smithi184 ceph-mon[102844]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:31.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:31.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:31.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:31 smithi196 ceph-mon[76349]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:56:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:33 smithi184 ceph-mon[100622]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:33 smithi184 ceph-mon[102844]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:33 smithi196 ceph-mon[76349]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:35 smithi184 ceph-mon[100622]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:35 smithi184 ceph-mon[102844]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:35 smithi196 ceph-mon[76349]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:37.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[100622]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:36 smithi184 ceph-mon[102844]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:36 smithi196 ceph-mon[76349]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.716 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:39 smithi184 ceph-mon[100622]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:39 smithi184 ceph-mon[102844]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:39 smithi196 ceph-mon[76349]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:41.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:41.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:41.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:41.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:41.475 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:41.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:41 smithi184 ceph-mon[100622]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:41 smithi184 ceph-mon[102844]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:41 smithi196 ceph-mon[76349]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:56:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:43 smithi184 ceph-mon[100622]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:43 smithi184 ceph-mon[102844]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:43 smithi196 ceph-mon[76349]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:44.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:45 smithi196 ceph-mon[76349]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:45 smithi184 ceph-mon[100622]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:45 smithi184 ceph-mon[102844]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:47.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:47 smithi184 ceph-mon[102844]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:47.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:47 smithi184 ceph-mon[100622]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:47 smithi196 ceph-mon[76349]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:49 smithi184 ceph-mon[102844]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:49 smithi184 ceph-mon[100622]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:49 smithi196 ceph-mon[76349]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:51.493 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:51.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:51.494 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:51.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:56:51.494 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:56:51.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:56:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:51 smithi184 ceph-mon[100622]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:51 smithi184 ceph-mon[102844]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:51 smithi196 ceph-mon[76349]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:56:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:56:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:53 smithi184 ceph-mon[100622]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:53 smithi184 ceph-mon[102844]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:53 smithi196 ceph-mon[76349]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:55 smithi184 ceph-mon[100622]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:55 smithi184 ceph-mon[102844]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:55 smithi196 ceph-mon[76349]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:56.534 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:56:57.073 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:56:57.073 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-20T23:56:57.073 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-20T23:56:57.073 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 1s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:56:57.073 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 1s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 1s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 1s ago 4h 421M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 1s ago 4h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 1s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 1s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 1s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 1s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:56:57.074 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 1s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 19s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 19s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 19s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 19s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 19s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:56:57.075 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 19s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[102844]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[102844]: from='client.70595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[100622]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:57.118 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:57 smithi184 ceph-mon[100622]: from='client.70595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:56:57.388 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:56:57.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:56:57.389 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:56:57.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:56:57.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:56:57.389 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:57 smithi196 ceph-mon[76349]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:57 smithi196 ceph-mon[76349]: from='client.70595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:57.666 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:56:57.666 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:56:57.666 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:56:57.666 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:56:57.667 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:56:57.983 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:56:57.984 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:57.984 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:57.984 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:57.984 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='client.70601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/875439534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3872723062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='client.70601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/875439534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:58.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3872723062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='client.70601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/875439534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:56:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3872723062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:59 smithi196 ceph-mon[76349]: from='client.70619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:59 smithi196 ceph-mon[76349]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:59.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[100622]: from='client.70619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:59.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:59.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[100622]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:59.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[102844]: from='client.70619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:59.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:59.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[102844]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:01 smithi184 ceph-mon[100622]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:01 smithi184 ceph-mon[102844]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:01.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:01.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:01.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:01 smithi196 ceph-mon[76349]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:57:03.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:03 smithi184 ceph-mon[100622]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:03.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:03 smithi184 ceph-mon[102844]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:03 smithi196 ceph-mon[76349]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:05.533 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:05 smithi196 ceph-mon[76349]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:05 smithi184 ceph-mon[100622]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:05 smithi184 ceph-mon[102844]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:07 smithi184 ceph-mon[100622]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:07 smithi184 ceph-mon[102844]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:07 smithi196 ceph-mon[76349]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:09 smithi184 ceph-mon[100622]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:09 smithi184 ceph-mon[102844]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:09 smithi196 ceph-mon[76349]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:11 smithi184 ceph-mon[100622]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:11 smithi184 ceph-mon[102844]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:11.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:11.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:11.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:11 smithi196 ceph-mon[76349]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:57:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:13 smithi184 ceph-mon[100622]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:13 smithi184 ceph-mon[102844]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:13 smithi196 ceph-mon[76349]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:15 smithi184 ceph-mon[100622]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:15 smithi184 ceph-mon[102844]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:15 smithi196 ceph-mon[76349]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[100622]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:17 smithi184 ceph-mon[102844]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:17 smithi196 ceph-mon[76349]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[100622]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:19.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:18 smithi184 ceph-mon[102844]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:18 smithi196 ceph-mon[76349]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:21.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:21.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:21.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:21.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:21.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:21.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:21 smithi184 ceph-mon[100622]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:21 smithi184 ceph-mon[102844]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:21 smithi196 ceph-mon[76349]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:57:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:23 smithi184 ceph-mon[100622]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:23 smithi184 ceph-mon[102844]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:23 smithi196 ceph-mon[76349]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:25 smithi196 ceph-mon[76349]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:25 smithi184 ceph-mon[100622]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:25 smithi184 ceph-mon[102844]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:27.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:27 smithi196 ceph-mon[76349]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:27 smithi184 ceph-mon[100622]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:27 smithi184 ceph-mon[102844]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:28.282 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:57:28.808 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 12s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 11s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 12s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 11s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 12s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 12s ago 4h 422M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 11s ago 4h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 12s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:57:28.809 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 11s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (3h) 12s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 12s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 12s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 12s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 11s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 11s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 12s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 12s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:57:28.810 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:57:29.120 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:57:29.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:57:29.122 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:57:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:57:29.389 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:57:29.695 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:57:29.695 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:29.695 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:29.695 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:29.695 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: from='client.60669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: from='client.70637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2159380552' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: from='client.60669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: from='client.70637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:29.696 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2159380552' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: from='client.60669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: from='client.70637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:29.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2159380552' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[100622]: from='client.70643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[100622]: from='client.70655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2091681795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[102844]: from='client.70643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[102844]: from='client.70655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2091681795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:30 smithi196 ceph-mon[76349]: from='client.70643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:30 smithi196 ceph-mon[76349]: from='client.70655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2091681795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:31.449 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:31.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:31.449 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:31.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:31.450 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:31.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:31 smithi184 ceph-mon[102844]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:31 smithi184 ceph-mon[100622]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:31 smithi196 ceph-mon[76349]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:57:33.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:33 smithi184 ceph-mon[100622]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:33.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:33 smithi184 ceph-mon[102844]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:33 smithi196 ceph-mon[76349]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:35 smithi184 ceph-mon[100622]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:35 smithi184 ceph-mon[102844]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:35 smithi196 ceph-mon[76349]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:37 smithi196 ceph-mon[76349]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[100622]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:37 smithi184 ceph-mon[102844]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[100622]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:38 smithi184 ceph-mon[102844]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:38 smithi196 ceph-mon[76349]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:41.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:41.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:41.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:41.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:41 smithi184 ceph-mon[100622]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:41 smithi184 ceph-mon[102844]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:41 smithi196 ceph-mon[76349]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:57:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:43 smithi184 ceph-mon[100622]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:43 smithi184 ceph-mon[102844]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:43 smithi196 ceph-mon[76349]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:45.771 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:45 smithi196 ceph-mon[76349]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:45 smithi184 ceph-mon[100622]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:45 smithi184 ceph-mon[102844]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:47 smithi196 ceph-mon[76349]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:47 smithi184 ceph-mon[100622]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:47 smithi184 ceph-mon[102844]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:49 smithi184 ceph-mon[102844]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:49 smithi184 ceph-mon[100622]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:49 smithi196 ceph-mon[76349]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:51.457 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:51.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:51.457 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:51.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:57:51.457 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:57:51.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:57:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:51 smithi184 ceph-mon[100622]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:51 smithi184 ceph-mon[102844]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:51 smithi196 ceph-mon[76349]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:57:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:57:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:53 smithi184 ceph-mon[100622]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:53 smithi184 ceph-mon[102844]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:53 smithi196 ceph-mon[76349]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:55 smithi184 ceph-mon[100622]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:55 smithi184 ceph-mon[102844]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:55 smithi196 ceph-mon[76349]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[100622]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[102844]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:57 smithi196 ceph-mon[76349]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:59.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:57:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:57:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:59.994 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 4s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 4s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:58:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 4s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 4s ago 4h 423M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 4s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 4s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 4s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 4s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 4s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 4s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 2s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 2s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 2s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 2s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:58:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:58:00.549 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:58:00.549 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 2s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:58:00.549 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:58:00.549 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 2s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:58:00.893 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:58:00.893 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:58:00.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:58:00.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:00.893 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:58:00.894 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:58:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:58:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:58:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:58:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:58:01.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:58:01.447 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:01.192Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:01.447 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:01.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:01.447 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:01.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:01.494 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:58:01.704 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:01.704 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:01.704 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:01.704 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[100622]: from='client.60705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[100622]: from='client.60711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[100622]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[100622]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1647567431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[102844]: from='client.60705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[102844]: from='client.60711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[102844]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[102844]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1647567431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:01 smithi196 ceph-mon[76349]: from='client.60705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:01 smithi196 ceph-mon[76349]: from='client.60711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:01 smithi196 ceph-mon[76349]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:01 smithi196 ceph-mon[76349]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1647567431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:58:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:02 smithi184 ceph-mon[100622]: from='client.70691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/479584931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:02 smithi184 ceph-mon[102844]: from='client.70691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/479584931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:02 smithi196 ceph-mon[76349]: from='client.70691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/479584931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:03 smithi184 ceph-mon[100622]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:03 smithi184 ceph-mon[102844]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:03 smithi196 ceph-mon[76349]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:05 smithi184 ceph-mon[100622]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:05 smithi184 ceph-mon[102844]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:05 smithi196 ceph-mon[76349]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:07 smithi196 ceph-mon[76349]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:07 smithi184 ceph-mon[100622]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:07 smithi184 ceph-mon[102844]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:09 smithi196 ceph-mon[76349]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:09 smithi184 ceph-mon[100622]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:09 smithi184 ceph-mon[102844]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:11.469 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:11.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:11.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:11.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:11.470 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:11.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:11 smithi184 ceph-mon[100622]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:11 smithi184 ceph-mon[102844]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:11 smithi196 ceph-mon[76349]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:58:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:13 smithi184 ceph-mon[102844]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:13 smithi184 ceph-mon[100622]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:13 smithi196 ceph-mon[76349]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 20 23:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-20T23:58:13.812990021Z level=info msg="Completed cleanup jobs" duration=12.914408ms 2024-09-20T23:58:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:15 smithi184 ceph-mon[100622]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:15 smithi184 ceph-mon[102844]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:15.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:15 smithi196 ceph-mon[76349]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:17.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[100622]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:17.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[102844]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:17.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:17 smithi196 ceph-mon[76349]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:19 smithi196 ceph-mon[76349]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[100622]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:19.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:19 smithi184 ceph-mon[102844]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:21.332 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:21.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:21.333 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:21.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:21.333 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:21.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:21.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:21 smithi184 ceph-mon[100622]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:21.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:21 smithi184 ceph-mon[102844]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:21 smithi196 ceph-mon[76349]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:22.244 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-20T23:58:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:23 smithi184 ceph-mon[100622]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:23 smithi184 ceph-mon[102844]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:23 smithi196 ceph-mon[76349]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:25.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:25 smithi196 ceph-mon[76349]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:25 smithi184 ceph-mon[100622]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:25 smithi184 ceph-mon[102844]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:27 smithi196 ceph-mon[76349]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:27 smithi184 ceph-mon[100622]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:27 smithi184 ceph-mon[102844]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[100622]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[102844]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:29 smithi196 ceph-mon[76349]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:31.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:31.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:31.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:31.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:31.456 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:31.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:31.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:31 smithi184 ceph-mon[100622]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:31.794 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:31 smithi184 ceph-mon[102844]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:31.794 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:58:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:31 smithi196 ceph-mon[76349]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 4h - - 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 15s ago 4h 424M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:58:32.337 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 15s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (3h) 15s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 15s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 15s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 14s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 14s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 74.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 15s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:58:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:58:32.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 15s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:58:32.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:58:32.683 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:58:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:58:32.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:58:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:58:32.967 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:58:33.282 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:58:33.283 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:33.283 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: from='client.60741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: from='client.60747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: from='client.70715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/275939373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2497971204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: from='client.60741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: from='client.60747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: from='client.70715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/275939373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:33.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2497971204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: from='client.60741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: from='client.60747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: from='client.70715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/275939373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2497971204' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:34 smithi184 ceph-mon[100622]: from='client.60765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:34 smithi184 ceph-mon[102844]: from='client.60765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:34 smithi196 ceph-mon[76349]: from='client.60765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:35 smithi184 ceph-mon[100622]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:35 smithi184 ceph-mon[102844]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:35 smithi196 ceph-mon[76349]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[102844]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[100622]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:37 smithi196 ceph-mon[76349]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[102844]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:39 smithi184 ceph-mon[100622]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:39 smithi196 ceph-mon[76349]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:41.460 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:41.195Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:41.460 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:41.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:41.460 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:41.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:41 smithi184 ceph-mon[100622]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:41 smithi184 ceph-mon[102844]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:41 smithi196 ceph-mon[76349]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:42.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-20T23:58:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:43 smithi184 ceph-mon[100622]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:43 smithi184 ceph-mon[102844]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:43 smithi196 ceph-mon[76349]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:45 smithi184 ceph-mon[100622]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:45 smithi184 ceph-mon[102844]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:45 smithi196 ceph-mon[76349]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:47 smithi196 ceph-mon[76349]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:47 smithi184 ceph-mon[100622]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:47 smithi184 ceph-mon[102844]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:49.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:49 smithi196 ceph-mon[76349]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:49 smithi184 ceph-mon[102844]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:49 smithi184 ceph-mon[100622]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:51.196Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:51.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:58:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:58:51.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:58:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:51 smithi184 ceph-mon[102844]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:51 smithi184 ceph-mon[100622]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:51 smithi196 ceph-mon[76349]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:58:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:58:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:53 smithi184 ceph-mon[100622]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:53 smithi184 ceph-mon[102844]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:53 smithi196 ceph-mon[76349]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:55 smithi184 ceph-mon[102844]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:55 smithi184 ceph-mon[100622]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:55 smithi196 ceph-mon[76349]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[100622]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[102844]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:57 smithi196 ceph-mon[76349]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:01.330 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:01.197Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:01.330 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:01.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:01.330 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:01.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:01.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:01 smithi184 ceph-mon[100622]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:01.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:01 smithi184 ceph-mon[102844]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:01 smithi196 ceph-mon[76349]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:02.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:59:03.575 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:59:03.711 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:03 smithi184 ceph-mon[100622]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:03.711 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:03 smithi184 ceph-mon[102844]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:03 smithi196 ceph-mon[76349]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 4h - - 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 5s ago 4h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 4h 422M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 5s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:59:04.142 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 5s ago 4h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (3h) 7s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 7s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 5s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 5s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 5s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 5s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 5s ago 4h 74.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 5s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:59:04.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 5s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:59:04.453 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:59:04.453 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:59:04.453 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:59:04.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:59:04.454 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:59:04.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:59:04.724 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:59:04.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:59:04.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:04 smithi184 ceph-mon[100622]: from='client.70739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:04.726 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3780700862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:04.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:04 smithi184 ceph-mon[102844]: from='client.70739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:04.726 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3780700862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:04 smithi196 ceph-mon[76349]: from='client.70739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3780700862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:05.036 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:59:05.036 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:05.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:05.036 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:05.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[100622]: from='client.70745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[100622]: from='client.70751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[100622]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/429804427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[102844]: from='client.70745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[102844]: from='client.70751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[102844]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/429804427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:05 smithi196 ceph-mon[76349]: from='client.70745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:05 smithi196 ceph-mon[76349]: from='client.70751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:05 smithi196 ceph-mon[76349]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/429804427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:06.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:06 smithi196 ceph-mon[76349]: from='client.70763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:06 smithi184 ceph-mon[100622]: from='client.70763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:06 smithi184 ceph-mon[102844]: from='client.70763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:07 smithi184 ceph-mon[102844]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:07 smithi184 ceph-mon[100622]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:07 smithi196 ceph-mon[76349]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:09.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:09.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:09 smithi196 ceph-mon[76349]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:09 smithi184 ceph-mon[102844]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:09 smithi184 ceph-mon[100622]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:11.490 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:11.198Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:11.490 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:11.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:11.491 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:11.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:11 smithi184 ceph-mon[100622]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:11 smithi184 ceph-mon[102844]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:11 smithi196 ceph-mon[76349]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:59:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:13 smithi184 ceph-mon[100622]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:13 smithi184 ceph-mon[102844]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:13 smithi196 ceph-mon[76349]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:15 smithi184 ceph-mon[100622]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:15 smithi184 ceph-mon[102844]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:15 smithi196 ceph-mon[76349]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[100622]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[102844]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:17 smithi196 ceph-mon[76349]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[100622]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[102844]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:19 smithi196 ceph-mon[76349]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:21.482 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:21.198Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:21.482 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:21.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:21.483 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:21.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:21 smithi184 ceph-mon[100622]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:21 smithi184 ceph-mon[102844]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:21 smithi196 ceph-mon[76349]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:22.102 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-20T23:59:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:23 smithi184 ceph-mon[100622]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:23 smithi184 ceph-mon[102844]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:23 smithi196 ceph-mon[76349]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:25 smithi184 ceph-mon[100622]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:25 smithi184 ceph-mon[102844]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:25 smithi196 ceph-mon[76349]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:27 smithi184 ceph-mon[100622]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:27 smithi184 ceph-mon[102844]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:27 smithi196 ceph-mon[76349]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:29.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:29 smithi196 ceph-mon[76349]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[102844]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[100622]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:31.496 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:31.199Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:31.497 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:31.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:31.497 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:31.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:31 smithi184 ceph-mon[100622]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:31 smithi184 ceph-mon[102844]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:31 smithi196 ceph-mon[76349]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:32] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:59:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:33 smithi184 ceph-mon[100622]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:33 smithi184 ceph-mon[102844]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:33 smithi196 ceph-mon[76349]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:35.333 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-20T23:59:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:35 smithi184 ceph-mon[100622]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:35 smithi184 ceph-mon[102844]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:35.863 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:59:35.863 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-20T23:59:35.863 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-20T23:59:35.863 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 4h 424M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (3h) 18s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-20T23:59:35.864 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 17s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 17s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 17s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 18s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 18s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-20T23:59:35.865 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-20T23:59:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:35 smithi196 ceph-mon[76349]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-20T23:59:36.177 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-20T23:59:36.178 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:59:36.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-20T23:59:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-20T23:59:36.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-20T23:59:36.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-20T23:59:36.757 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:59:36.757 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:36.757 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:36.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[102844]: from='client.60813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[102844]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/377553348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:36.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[100622]: from='client.60813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[100622]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/377553348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:36 smithi196 ceph-mon[76349]: from='client.60813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:36 smithi196 ceph-mon[76349]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/377553348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='client.70787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='client.70799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/275108229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='client.70787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='client.70799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/275108229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='client.70787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='client.70799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/275108229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:39 smithi196 ceph-mon[76349]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[102844]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[100622]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:41.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:41.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:41.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:41 smithi196 ceph-mon[76349]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:41 smithi184 ceph-mon[100622]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:41 smithi184 ceph-mon[102844]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:42] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-20T23:59:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:43 smithi196 ceph-mon[76349]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:43.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:43 smithi184 ceph-mon[100622]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:43.893 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:43 smithi184 ceph-mon[102844]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:45 smithi196 ceph-mon[76349]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:45 smithi184 ceph-mon[100622]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:45 smithi184 ceph-mon[102844]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:47 smithi196 ceph-mon[76349]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:47.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:47 smithi184 ceph-mon[100622]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:47.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:47 smithi184 ceph-mon[102844]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:48 smithi184 ceph-mon[102844]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:48 smithi184 ceph-mon[100622]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:49.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:49.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:48 smithi196 ceph-mon[76349]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:51.201Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:51.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-20T23:59:51.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 20 23:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-20T23:59:51.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-20T23:59:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:51 smithi184 ceph-mon[100622]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:51 smithi184 ceph-mon[102844]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:51 smithi196 ceph-mon[76349]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 20 23:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [20/Sep/2024:23:59:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-20T23:59:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:53 smithi184 ceph-mon[100622]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:53 smithi184 ceph-mon[102844]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:53 smithi196 ceph-mon[76349]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:55 smithi184 ceph-mon[100622]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:55 smithi184 ceph-mon[102844]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:55 smithi196 ceph-mon[76349]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[100622]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[102844]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:59:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:57 smithi196 ceph-mon[76349]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:59:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 20 23:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-20T23:59:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 20 23:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-20T23:59:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:01.506 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:01.201Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:01.506 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:01.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:01.507 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:01.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:01 smithi184 ceph-mon[100622]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:01 smithi184 ceph-mon[102844]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:01 smithi196 ceph-mon[76349]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:00:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:03 smithi184 ceph-mon[100622]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:00:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:03 smithi184 ceph-mon[102844]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:00:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:03 smithi196 ceph-mon[76349]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:00:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:05 smithi184 ceph-mon[102844]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T00:00:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:05 smithi184 ceph-mon[100622]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T00:00:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:05 smithi196 ceph-mon[76349]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T00:00:07.040 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 4h - - 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 4h - - 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 10s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 10s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 10s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:00:07.569 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 10s ago 4h 425M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 8s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 10s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 10s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 10s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 10s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 4h 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 8s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 8s ago 4h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 8s ago 4h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 8s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 74.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:00:07.570 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 10s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:00:07.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 8s ago 4h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:00:07.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 10s ago 4h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:00:07.571 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 8s ago 4h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:00:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:07 smithi184 ceph-mon[100622]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:07 smithi184 ceph-mon[102844]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:07.877 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:00:07.877 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:00:07.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:00:07.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:07.877 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:00:07.878 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:00:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:07 smithi196 ceph-mon[76349]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:00:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:00:08.150 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:00:08.150 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:00:08.460 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:08.460 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:08.460 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:08.460 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:08.460 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[100622]: from='client.60849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[100622]: from='client.60855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[100622]: from='client.70823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/860947215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2459735698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[102844]: from='client.60849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[102844]: from='client.60855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[102844]: from='client.70823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/860947215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2459735698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:08 smithi196 ceph-mon[76349]: from='client.60849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:08 smithi196 ceph-mon[76349]: from='client.60855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:08 smithi196 ceph-mon[76349]: from='client.70823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/860947215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2459735698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[102844]: from='client.60873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[102844]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[100622]: from='client.60873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:09 smithi184 ceph-mon[100622]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:09.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:09.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:09 smithi196 ceph-mon[76349]: from='client.60873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:09.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:09 smithi196 ceph-mon[76349]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:11.527 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:11.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:11.527 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:11.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:11.527 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:11.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:11.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:11 smithi196 ceph-mon[76349]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:11 smithi184 ceph-mon[102844]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:11 smithi184 ceph-mon[100622]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:00:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:13 smithi184 ceph-mon[102844]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:13 smithi184 ceph-mon[100622]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:13 smithi196 ceph-mon[76349]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:15 smithi184 ceph-mon[100622]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:00:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:15 smithi184 ceph-mon[102844]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:00:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:15 smithi196 ceph-mon[76349]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:00:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[100622]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:00:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[102844]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:00:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:17 smithi196 ceph-mon[76349]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:00:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:19.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[100622]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:19.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[102844]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:19 smithi196 ceph-mon[76349]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:19.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:21.545 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:21.203Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:21.546 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:21.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:21.546 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:21.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:21 smithi184 ceph-mon[100622]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:21 smithi184 ceph-mon[102844]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:21 smithi196 ceph-mon[76349]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-21T00:00:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:23 smithi196 ceph-mon[76349]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:23 smithi184 ceph-mon[102844]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:23 smithi184 ceph-mon[100622]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:25 smithi196 ceph-mon[76349]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:26.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:25 smithi184 ceph-mon[100622]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:26.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:25 smithi184 ceph-mon[102844]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:27 smithi196 ceph-mon[76349]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:27.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:27 smithi184 ceph-mon[100622]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:27.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:27 smithi184 ceph-mon[102844]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:29 smithi196 ceph-mon[76349]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[100622]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[102844]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:31.204Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:31.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:31.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:31.821 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:31 smithi196 ceph-mon[76349]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:31 smithi184 ceph-mon[100622]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:31 smithi184 ceph-mon[102844]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:00:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:33 smithi196 ceph-mon[76349]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:33 smithi184 ceph-mon[100622]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:33 smithi184 ceph-mon[102844]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:35 smithi196 ceph-mon[76349]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:35 smithi184 ceph-mon[100622]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:35 smithi184 ceph-mon[102844]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[102844]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[100622]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:37 smithi196 ceph-mon[76349]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:38.756 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 1s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 1s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 1s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 1s ago 4h 427M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:00:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 1s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 1s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 1s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 1s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 1s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 1s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (3h) 0s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 0s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 0s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 0s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 1s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:00:39.287 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:00:39.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 1s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:00:39.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:39.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:00:39.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[100622]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[102844]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:00:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:00:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:00:39.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:00:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:39 smithi196 ceph-mon[76349]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:40.179 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:40.179 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:40.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:40.179 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:40.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:40 smithi196 ceph-mon[76349]: from='client.60885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:40 smithi196 ceph-mon[76349]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:40 smithi196 ceph-mon[76349]: from='client.70859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/39859404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4131831555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[100622]: from='client.60885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[100622]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[100622]: from='client.70859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/39859404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4131831555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[102844]: from='client.60885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[102844]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[102844]: from='client.70859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/39859404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4131831555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:41.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:41.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:41.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:41 smithi196 ceph-mon[76349]: from='client.60909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:41 smithi196 ceph-mon[76349]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-mon[100622]: from='client.60909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-mon[100622]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-mon[102844]: from='client.60909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:41 smithi184 ceph-mon[102844]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:00:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:43 smithi196 ceph-mon[76349]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:43 smithi184 ceph-mon[100622]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:43 smithi184 ceph-mon[102844]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:44.893 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:44.893 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:45.776 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:45 smithi184 ceph-mon[100622]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:45.776 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:45 smithi184 ceph-mon[102844]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:45 smithi196 ceph-mon[76349]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:47.867 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:47 smithi184 ceph-mon[100622]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:47.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:47 smithi184 ceph-mon[102844]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:47 smithi196 ceph-mon[76349]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:49 smithi196 ceph-mon[76349]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:49 smithi184 ceph-mon[100622]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:49 smithi184 ceph-mon[102844]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:51.206Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:51.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:00:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:00:51.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:00:51.883 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:51 smithi196 ceph-mon[76349]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:51 smithi184 ceph-mon[100622]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:51 smithi184 ceph-mon[102844]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:00:52] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-21T00:00:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:53 smithi196 ceph-mon[76349]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:53 smithi184 ceph-mon[100622]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:53 smithi184 ceph-mon[102844]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:55 smithi196 ceph-mon[76349]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:55 smithi184 ceph-mon[100622]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:55 smithi184 ceph-mon[102844]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:57 smithi196 ceph-mon[76349]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:00:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[100622]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[102844]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:58.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:00:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:01:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:01.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:01.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:01.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:01 smithi196 ceph-mon[76349]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:01 smithi184 ceph-mon[100622]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:01 smithi184 ceph-mon[102844]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:02] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:01:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:03 smithi196 ceph-mon[76349]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:01:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:03 smithi184 ceph-mon[100622]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:01:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:03 smithi184 ceph-mon[102844]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:01:05.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:05 smithi184 ceph-mon[100622]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:05.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:05 smithi184 ceph-mon[102844]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:05 smithi196 ceph-mon[76349]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:07.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:07 smithi184 ceph-mon[100622]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:07.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:07 smithi184 ceph-mon[102844]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:07 smithi196 ceph-mon[76349]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:09 smithi196 ceph-mon[76349]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:09 smithi184 ceph-mon[102844]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:09 smithi184 ceph-mon[100622]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:10.476 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 4h - - 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 13s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 11s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 13s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 11s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 13s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 13s ago 4h 427M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:01:11.001 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 11s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 13s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 13s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 11s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 13s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 13s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 13s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 13s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 11s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 11s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 11s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 11s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 11s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 13s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:01:11.002 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 13s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:01:11.003 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 11s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:01:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:11.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:11.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:11.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:11.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:11.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:01:11.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:01:11.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:01:11.315 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:01:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:01:11.587 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:01:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:11 smithi196 ceph-mon[76349]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:11 smithi196 ceph-mon[76349]: from='client.60921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/784022641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:11.897 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:01:11.897 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:11.897 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:11.898 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[100622]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[100622]: from='client.60921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/784022641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[102844]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[102844]: from='client.60921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:11.898 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/784022641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:12] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:12 smithi196 ceph-mon[76349]: from='client.70889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:12 smithi196 ceph-mon[76349]: from='client.70895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:12 smithi196 ceph-mon[76349]: from='client.60945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2288081065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[100622]: from='client.70889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[100622]: from='client.70895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[100622]: from='client.60945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2288081065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[102844]: from='client.70889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[102844]: from='client.70895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[102844]: from='client.60945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2288081065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:13 smithi196 ceph-mon[76349]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:13 smithi184 ceph-mon[100622]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:13 smithi184 ceph-mon[102844]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:15 smithi184 ceph-mon[100622]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:15 smithi184 ceph-mon[102844]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:15 smithi196 ceph-mon[76349]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:16 smithi184 ceph-mon[100622]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:16 smithi184 ceph-mon[102844]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:16 smithi196 ceph-mon[76349]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[100622]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:18 smithi184 ceph-mon[102844]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:18 smithi196 ceph-mon[76349]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:21 smithi184 ceph-mon[100622]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:21 smithi184 ceph-mon[102844]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:21.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:21.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:21.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:21 smithi196 ceph-mon[76349]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:22] "GET /metrics HTTP/1.1" 200 38937 "" "Prometheus/2.43.0" 2024-09-21T00:01:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:23 smithi184 ceph-mon[100622]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:23 smithi184 ceph-mon[102844]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:23 smithi196 ceph-mon[76349]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:25 smithi184 ceph-mon[100622]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:25 smithi184 ceph-mon[102844]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:25 smithi196 ceph-mon[76349]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:27.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:27 smithi184 ceph-mon[100622]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:27.712 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:27 smithi184 ceph-mon[102844]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:27 smithi196 ceph-mon[76349]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[100622]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[102844]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:29 smithi196 ceph-mon[76349]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:31.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:31.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:31.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:31.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:31.526 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:31.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:31 smithi184 ceph-mon[102844]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:31 smithi184 ceph-mon[100622]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:31 smithi196 ceph-mon[76349]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:32] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-21T00:01:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:33 smithi184 ceph-mon[100622]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:33 smithi184 ceph-mon[102844]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:33.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:33 smithi196 ceph-mon[76349]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:35 smithi184 ceph-mon[100622]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:35 smithi184 ceph-mon[102844]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:35 smithi196 ceph-mon[76349]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:37 smithi184 ceph-mon[100622]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:37 smithi184 ceph-mon[102844]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:37 smithi196 ceph-mon[76349]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:38 smithi196 ceph-mon[76349]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[100622]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:38 smithi184 ceph-mon[102844]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:41 smithi184 ceph-mon[100622]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:41 smithi184 ceph-mon[102844]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:41.209Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:41.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:41.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:41 smithi196 ceph-mon[76349]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:42.191 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:01:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:42] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-21T00:01:42.719 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 4h - - 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 4s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 3s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 4s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 3s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 4s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 4s ago 4h 428M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 3s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 4s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 4s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 3s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:01:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 4s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 4s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 4s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 4s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (3h) 3s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 3s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 3s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 3s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 4s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 4s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:01:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 3s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:01:43.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:01:43.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:01:43.033 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:01:43.033 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:01:43.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:01:43.033 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:01:43.297 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:01:43.298 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:01:43.299 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:01:43.299 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:01:43.299 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:01:43.299 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[100622]: from='client.70919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[100622]: from='client.60963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[100622]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1759214145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[102844]: from='client.70919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[102844]: from='client.60963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[102844]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1759214145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:43.613 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:01:43.613 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:43.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:43.613 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:43.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:43 smithi196 ceph-mon[76349]: from='client.70919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:43 smithi196 ceph-mon[76349]: from='client.60963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:43 smithi196 ceph-mon[76349]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1759214145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[100622]: from='client.70931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[100622]: from='client.70943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2953512331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[102844]: from='client.70931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[102844]: from='client.70943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2953512331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:44 smithi196 ceph-mon[76349]: from='client.70931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:44 smithi196 ceph-mon[76349]: from='client.70943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2953512331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:45 smithi184 ceph-mon[100622]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:45 smithi184 ceph-mon[102844]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:45 smithi196 ceph-mon[76349]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:47 smithi184 ceph-mon[100622]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:47.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:47 smithi184 ceph-mon[102844]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:47 smithi196 ceph-mon[76349]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:49 smithi184 ceph-mon[100622]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:49 smithi184 ceph-mon[102844]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:49 smithi196 ceph-mon[76349]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:51.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:51.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:01:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:01:51.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:01:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:51 smithi196 ceph-mon[76349]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:51 smithi184 ceph-mon[102844]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:51 smithi184 ceph-mon[100622]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:01:52] "GET /metrics HTTP/1.1" 200 38930 "" "Prometheus/2.43.0" 2024-09-21T00:01:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:53 smithi196 ceph-mon[76349]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:53 smithi184 ceph-mon[100622]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:53 smithi184 ceph-mon[102844]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:55 smithi196 ceph-mon[76349]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:55 smithi184 ceph-mon[100622]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:55 smithi184 ceph-mon[102844]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:57 smithi196 ceph-mon[76349]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:57.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:57 smithi184 ceph-mon[100622]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:57.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:57 smithi184 ceph-mon[102844]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[100622]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:58 smithi184 ceph-mon[102844]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:58 smithi196 ceph-mon[76349]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:00 smithi184 ceph-mon[100622]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:00 smithi184 ceph-mon[102844]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:01.211Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:01.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:01.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:01.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:00 smithi196 ceph-mon[76349]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:02] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:03 smithi184 ceph-mon[100622]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:03 smithi184 ceph-mon[102844]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:03 smithi196 ceph-mon[76349]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:05 smithi184 ceph-mon[100622]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:05 smithi184 ceph-mon[102844]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:05 smithi196 ceph-mon[76349]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:07 smithi196 ceph-mon[76349]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:07 smithi184 ceph-mon[100622]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:07 smithi184 ceph-mon[102844]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:09.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:09.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:09 smithi184 ceph-mon[100622]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:09.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:09.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:09 smithi184 ceph-mon[102844]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:09 smithi196 ceph-mon[76349]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:11.541 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:11.211Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:11.541 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:11.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:11.541 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:11.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:11 smithi184 ceph-mon[100622]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:11 smithi184 ceph-mon[102844]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:11 smithi196 ceph-mon[76349]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:12] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:13 smithi184 ceph-mon[100622]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:13 smithi184 ceph-mon[102844]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:13 smithi196 ceph-mon[76349]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.910 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:02:14.444 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:02:14.444 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-21T00:02:14.444 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 16s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 16s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 16s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 16s ago 4h 432M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 16s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:02:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 16s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 16s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 16s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 16s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 14s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 14s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 16s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:02:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:02:14.447 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 16s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:02:14.447 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 14s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:02:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:14.762 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:02:14.762 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:02:14.762 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:02:14.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:02:14.763 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:02:14.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:14.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:15.029 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:02:15.029 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:02:15.029 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:02:15.030 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:02:15.030 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:02:15.030 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:02:15.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:02:15.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:02:15.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:02:15.341 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:02:15.341 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:15.341 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:15.341 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:15.341 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:15.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: from='client.70955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: from='client.70961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:15.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3645484529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:15.600 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1030836181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: from='client.70955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: from='client.70961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3645484529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1030836181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: from='client.70955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: from='client.70961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3645484529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1030836181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:16 smithi196 ceph-mon[76349]: from='client.61017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:16 smithi184 ceph-mon[100622]: from='client.61017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:16 smithi184 ceph-mon[102844]: from='client.61017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:17 smithi196 ceph-mon[76349]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:17 smithi184 ceph-mon[100622]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:17 smithi184 ceph-mon[102844]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.351 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.351 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.351 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.351 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:19.351 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[102844]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:19.352 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:19 smithi184 ceph-mon[100622]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:19 smithi196 ceph-mon[76349]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:21.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:21.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:21.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:21 smithi184 ceph-mon[100622]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:21 smithi184 ceph-mon[102844]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:21 smithi196 ceph-mon[76349]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:23 smithi184 ceph-mon[100622]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:23 smithi184 ceph-mon[102844]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:23 smithi196 ceph-mon[76349]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:24 smithi196 ceph-mon[76349]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:24 smithi184 ceph-mon[100622]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:24 smithi184 ceph-mon[102844]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:27 smithi184 ceph-mon[102844]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:27 smithi184 ceph-mon[100622]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:27 smithi196 ceph-mon[76349]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:29.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[102844]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:29.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:29.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[100622]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:29 smithi196 ceph-mon[76349]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:31.539 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:31.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:31.539 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:31.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:31.539 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:31.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:31 smithi184 ceph-mon[100622]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:31 smithi184 ceph-mon[102844]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:31 smithi196 ceph-mon[76349]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:32] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:33 smithi184 ceph-mon[100622]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:33 smithi184 ceph-mon[102844]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:33 smithi196 ceph-mon[76349]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:35 smithi196 ceph-mon[76349]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:35 smithi184 ceph-mon[100622]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:35 smithi184 ceph-mon[102844]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:37 smithi184 ceph-mon[102844]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:37.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:37 smithi184 ceph-mon[100622]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:37 smithi196 ceph-mon[76349]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[100622]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:39 smithi184 ceph-mon[102844]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:39 smithi196 ceph-mon[76349]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[100622]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:40 smithi184 ceph-mon[102844]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:40 smithi196 ceph-mon[76349]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:41.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:41.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:41.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:42] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:43 smithi184 ceph-mon[100622]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:43.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:43 smithi184 ceph-mon[102844]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:43 smithi196 ceph-mon[76349]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:45.640 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:02:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:45 smithi196 ceph-mon[76349]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:45 smithi184 ceph-mon[100622]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:45 smithi184 ceph-mon[102844]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:02:46.170 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 4h 429M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (3h) 6s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 6s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 7s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:02:46.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:02:46.172 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 7s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:02:46.172 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:02:46.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:02:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:02:46.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:02:46.479 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:02:46.749 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:02:46.749 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:02:46.749 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:02:46.749 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:02:46.750 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:02:46.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:46 smithi184 ceph-mon[100622]: from='client.70991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:46.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/683144068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:46 smithi184 ceph-mon[102844]: from='client.70991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:46.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/683144068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:46 smithi196 ceph-mon[76349]: from='client.70991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/683144068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:47.062 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:02:47.062 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:47.062 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:47.063 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:47 smithi196 ceph-mon[76349]: from='client.70997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:47 smithi196 ceph-mon[76349]: from='client.71003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:47 smithi196 ceph-mon[76349]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2881694539' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[100622]: from='client.70997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[100622]: from='client.71003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[100622]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2881694539' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[102844]: from='client.70997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[102844]: from='client.71003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[102844]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2881694539' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:48.877 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:48 smithi184 ceph-mon[100622]: from='client.71015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:48 smithi184 ceph-mon[102844]: from='client.71015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:48 smithi196 ceph-mon[76349]: from='client.71015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:49 smithi196 ceph-mon[76349]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:49.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:49.953 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:49 smithi184 ceph-mon[100622]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:49.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:49.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:49 smithi184 ceph-mon[102844]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:51.214Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:51.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:02:51.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:02:51.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:02:51.821 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:51 smithi184 ceph-mon[100622]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:51.822 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:51 smithi184 ceph-mon[102844]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:51 smithi196 ceph-mon[76349]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:52.266 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:02:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:02:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:53 smithi196 ceph-mon[76349]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:53 smithi184 ceph-mon[100622]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:53 smithi184 ceph-mon[102844]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:55 smithi196 ceph-mon[76349]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:55 smithi184 ceph-mon[100622]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:55 smithi184 ceph-mon[102844]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:57 smithi196 ceph-mon[76349]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:57 smithi184 ceph-mon[100622]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:57 smithi184 ceph-mon[102844]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:01.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[100622]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:01.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:00 smithi184 ceph-mon[102844]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:00 smithi196 ceph-mon[76349]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:01.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:01.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:01.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:02] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-21T00:03:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:03 smithi196 ceph-mon[76349]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:03 smithi184 ceph-mon[100622]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:03 smithi184 ceph-mon[102844]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:05 smithi196 ceph-mon[76349]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:05 smithi184 ceph-mon[100622]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:05 smithi184 ceph-mon[102844]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:07 smithi196 ceph-mon[76349]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:07 smithi184 ceph-mon[100622]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:07 smithi184 ceph-mon[102844]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:09.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:09.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:09 smithi184 ceph-mon[100622]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:09.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:09.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:09 smithi184 ceph-mon[102844]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:09 smithi196 ceph-mon[76349]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:11.463 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:11.216Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:11.463 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:11.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:11.463 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:11.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:11 smithi184 ceph-mon[100622]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:11 smithi184 ceph-mon[102844]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:11 smithi196 ceph-mon[76349]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:12.066 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:12] "GET /metrics HTTP/1.1" 200 38933 "" "Prometheus/2.43.0" 2024-09-21T00:03:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:13 smithi196 ceph-mon[76349]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:13 smithi184 ceph-mon[100622]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:13 smithi184 ceph-mon[102844]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:15 smithi196 ceph-mon[76349]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:15 smithi184 ceph-mon[100622]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:15 smithi184 ceph-mon[102844]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:17.360 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:03:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:17 smithi184 ceph-mon[100622]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:17 smithi184 ceph-mon[102844]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:17.889 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:03:17.889 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-21T00:03:17.889 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-21T00:03:17.889 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 4h 429M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:03:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (3h) 17s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 19s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 19s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:03:17.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 17s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:03:17.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:17 smithi196 ceph-mon[76349]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:03:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:03:18.202 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:03:18.490 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:03:18.490 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:03:18.490 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:03:18.490 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:03:18.490 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:03:18.491 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='client.71027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='client.71033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4199706517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='client.71027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='client.71033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4199706517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.802 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:03:18.802 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:18.802 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:18.802 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='client.71027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='client.71033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4199706517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:19 smithi196 ceph-mon[76349]: from='client.61077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:19 smithi196 ceph-mon[76349]: from='client.71051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:19 smithi196 ceph-mon[76349]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2529316981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[100622]: from='client.61077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[100622]: from='client.71051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[100622]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2529316981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[102844]: from='client.61077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[102844]: from='client.71051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[102844]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2529316981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[100622]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-mon[102844]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:21.216Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:21.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:21.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:21.301 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:21.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:21 smithi196 ceph-mon[76349]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:22] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:03:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:23 smithi196 ceph-mon[76349]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:23 smithi184 ceph-mon[100622]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:23 smithi184 ceph-mon[102844]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:25 smithi196 ceph-mon[76349]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:25 smithi184 ceph-mon[100622]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:25 smithi184 ceph-mon[102844]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:27 smithi196 ceph-mon[76349]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:27 smithi184 ceph-mon[100622]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:27 smithi184 ceph-mon[102844]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:29 smithi196 ceph-mon[76349]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[100622]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[102844]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:31.537 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:31.217Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:31.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:31.218Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:31.538 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:31.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:31 smithi184 ceph-mon[100622]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:31 smithi184 ceph-mon[102844]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:31 smithi196 ceph-mon[76349]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:32.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:03:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:33 smithi196 ceph-mon[76349]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:33 smithi184 ceph-mon[100622]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:33 smithi184 ceph-mon[102844]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:35 smithi196 ceph-mon[76349]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:35 smithi184 ceph-mon[100622]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:35 smithi184 ceph-mon[102844]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:37.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:37 smithi196 ceph-mon[76349]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:37 smithi184 ceph-mon[100622]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:37 smithi184 ceph-mon[102844]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:39 smithi196 ceph-mon[76349]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[100622]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[102844]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[100622]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-mon[102844]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:41.218Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:41.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:41.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:41 smithi196 ceph-mon[76349]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:03:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:43 smithi184 ceph-mon[100622]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:43 smithi184 ceph-mon[102844]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:43 smithi196 ceph-mon[76349]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:45 smithi196 ceph-mon[76349]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:45 smithi184 ceph-mon[100622]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:45 smithi184 ceph-mon[102844]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:47 smithi196 ceph-mon[76349]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:47 smithi184 ceph-mon[100622]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:47 smithi184 ceph-mon[102844]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.098 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 9s ago 4h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:03:49.629 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 4h 430M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 9s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 9s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 4h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:03:49.630 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 4h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 9s ago 4h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 9s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 9s ago 4h 75.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (3h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (3h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (3h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:03:49.631 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 9s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:03:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:49 smithi196 ceph-mon[76349]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:49.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:49 smithi184 ceph-mon[100622]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.941 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:49.941 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:49 smithi184 ceph-mon[102844]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.941 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:03:49.942 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:03:49.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:03:49.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:03:49.943 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:03:50.209 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:03:50.210 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:03:50.520 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:03:50.520 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:50.520 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:50.520 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:50.520 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:50.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[100622]: from='client.71063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[100622]: from='client.61107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[100622]: from='client.71075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2593108493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:50.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3936263076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:50.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[102844]: from='client.71063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[102844]: from='client.61107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[102844]: from='client.71075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2593108493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:50.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3936263076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:50 smithi196 ceph-mon[76349]: from='client.71063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:50 smithi196 ceph-mon[76349]: from='client.61107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:50 smithi196 ceph-mon[76349]: from='client.71075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2593108493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3936263076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:51.219Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:51.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:03:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:03:51.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:03:51.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-mon[100622]: from='client.61125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:51.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-mon[100622]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:51.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-mon[102844]: from='client.61125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:51.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:51 smithi184 ceph-mon[102844]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:51 smithi196 ceph-mon[76349]: from='client.61125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:51 smithi196 ceph-mon[76349]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:52.138 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:03:52] "GET /metrics HTTP/1.1" 200 38934 "" "Prometheus/2.43.0" 2024-09-21T00:03:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:53 smithi196 ceph-mon[76349]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:53 smithi184 ceph-mon[100622]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:53 smithi184 ceph-mon[102844]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:55 smithi196 ceph-mon[76349]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:55 smithi184 ceph-mon[100622]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:55 smithi184 ceph-mon[102844]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:57 smithi196 ceph-mon[76349]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:57 smithi184 ceph-mon[100622]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:57 smithi184 ceph-mon[102844]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:03:59.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[100622]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-mon[102844]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:01.220Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:01.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:01.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:01 smithi196 ceph-mon[76349]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:02] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-21T00:04:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:03 smithi184 ceph-mon[100622]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:03 smithi184 ceph-mon[102844]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:03 smithi196 ceph-mon[76349]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:05 smithi196 ceph-mon[76349]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T00:04:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:05 smithi184 ceph-mon[100622]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T00:04:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:05 smithi184 ceph-mon[102844]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T00:04:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:07 smithi196 ceph-mon[76349]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:04:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:07 smithi184 ceph-mon[102844]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:04:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:07 smithi184 ceph-mon[100622]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:04:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:09 smithi196 ceph-mon[76349]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T00:04:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:09 smithi184 ceph-mon[102844]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T00:04:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:09 smithi184 ceph-mon[100622]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T00:04:11.522 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:11.221Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:11.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:11.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:11.523 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:11.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:11 smithi184 ceph-mon[100622]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T00:04:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:11 smithi184 ceph-mon[102844]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T00:04:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:11 smithi196 ceph-mon[76349]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T00:04:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:12] "GET /metrics HTTP/1.1" 200 38936 "" "Prometheus/2.43.0" 2024-09-21T00:04:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:13 smithi196 ceph-mon[76349]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T00:04:13.902 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:13 smithi184 ceph-mon[100622]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T00:04:13.903 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:13 smithi184 ceph-mon[102844]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T00:04:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:15 smithi196 ceph-mon[76349]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:04:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:15 smithi184 ceph-mon[100622]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:04:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:15 smithi184 ceph-mon[102844]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:04:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:17 smithi196 ceph-mon[76349]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:04:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:17 smithi184 ceph-mon[100622]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:04:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:17 smithi184 ceph-mon[102844]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:04:19.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:19.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:19 smithi196 ceph-mon[76349]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:19.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:19.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:19.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[102844]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[100622]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:20.815 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 4h - - 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 4h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 4h 431M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:04:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 4h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 4h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 4h 86.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 4h 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 4h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 0s ago 4h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 0s ago 4h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 0s ago 4h 86.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 4h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:04:21.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:04:21.342 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:04:21.342 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (3h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[100622]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-mon[102844]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T00:04:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:21.221Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:21.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:21.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:21.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:21.651 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:04:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:04:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:04:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:04:21.653 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:04:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:21 smithi196 ceph-mon[76349]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:04:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:04:21.921 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:04:21.921 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:04:21.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:04:22.229 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:04:22.229 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:22.229 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:22.229 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:22.229 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:22.229 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[100622]: from='client.61137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[100622]: from='client.61143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[100622]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3923510290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4009040919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[102844]: from='client.61137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[102844]: from='client.61143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[102844]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3923510290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4009040919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:22 smithi196 ceph-mon[76349]: from='client.61137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:22 smithi196 ceph-mon[76349]: from='client.61143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:22 smithi196 ceph-mon[76349]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3923510290' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4009040919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:23 smithi184 ceph-mon[100622]: from='client.71123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:23 smithi184 ceph-mon[100622]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:04:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:23 smithi184 ceph-mon[102844]: from='client.71123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:23 smithi184 ceph-mon[102844]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:04:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:23 smithi196 ceph-mon[76349]: from='client.71123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:23 smithi196 ceph-mon[76349]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:04:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:25 smithi196 ceph-mon[76349]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T00:04:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:25 smithi184 ceph-mon[100622]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T00:04:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:25 smithi184 ceph-mon[102844]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T00:04:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:27 smithi196 ceph-mon[76349]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T00:04:28.343 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:27 smithi184 ceph-mon[100622]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T00:04:28.343 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:27 smithi184 ceph-mon[102844]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T00:04:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:29 smithi196 ceph-mon[76349]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-21T00:04:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[100622]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[102844]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-21T00:04:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:31.222Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:31.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:31.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:31 smithi196 ceph-mon[76349]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:04:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:31 smithi184 ceph-mon[100622]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:04:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:31 smithi184 ceph-mon[102844]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:04:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:32] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:04:33.675 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:33 smithi184 ceph-mon[100622]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:04:33.676 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:33 smithi184 ceph-mon[102844]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:04:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:33 smithi196 ceph-mon[76349]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:04:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:35 smithi196 ceph-mon[76349]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T00:04:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:35 smithi184 ceph-mon[100622]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T00:04:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:35 smithi184 ceph-mon[102844]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T00:04:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:37 smithi196 ceph-mon[76349]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:04:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:37 smithi184 ceph-mon[102844]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:04:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:37 smithi184 ceph-mon[100622]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:04:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:39 smithi196 ceph-mon[76349]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[100622]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[102844]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:04:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.531 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:41.223Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:41.531 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:41.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:41.531 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:41.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[100622]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:04:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[102844]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:04:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:41 smithi196 ceph-mon[76349]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:04:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:42] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:04:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:43 smithi184 ceph-mon[100622]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:04:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:43 smithi184 ceph-mon[102844]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:04:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:43 smithi196 ceph-mon[76349]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:04:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:45 smithi184 ceph-mon[100622]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:04:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:45 smithi184 ceph-mon[102844]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:04:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:45 smithi196 ceph-mon[76349]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:04:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:47 smithi196 ceph-mon[76349]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:04:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:47 smithi184 ceph-mon[100622]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:04:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:47 smithi184 ceph-mon[102844]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:04:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:49 smithi196 ceph-mon[76349]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:49 smithi184 ceph-mon[100622]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:49 smithi184 ceph-mon[102844]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:51.224Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:51.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:04:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:04:51.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:04:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:51 smithi196 ceph-mon[76349]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:51 smithi184 ceph-mon[100622]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:51 smithi184 ceph-mon[102844]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:04:52] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-21T00:04:52.525 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:04:53.084 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:04:53.084 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 4h 432M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:04:53.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 4h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 4h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 4h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 4h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 4h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 4h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 4h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:04:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:04:53.405 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:04:53.405 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:04:53.405 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:53.406 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:04:53.407 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:04:53.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:04:53.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[100622]: from='client.61173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[100622]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/384889270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[102844]: from='client.61173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[102844]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:53.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/384889270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:53 smithi196 ceph-mon[76349]: from='client.61173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:53 smithi196 ceph-mon[76349]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/384889270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:54.012 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:04:54.012 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:54.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:54.012 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:54.012 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:54 smithi196 ceph-mon[76349]: from='client.71141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:54 smithi196 ceph-mon[76349]: from='client.61185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4188536485' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[102844]: from='client.71141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[102844]: from='client.61185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4188536485' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[100622]: from='client.71141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[100622]: from='client.61185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:54.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4188536485' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:55.873 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:55 smithi184 ceph-mon[100622]: from='client.71159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:55.873 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:55 smithi184 ceph-mon[100622]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:55.873 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:55 smithi184 ceph-mon[102844]: from='client.71159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:55.874 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:55 smithi184 ceph-mon[102844]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:55 smithi196 ceph-mon[76349]: from='client.71159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:55 smithi196 ceph-mon[76349]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:57 smithi196 ceph-mon[76349]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:57 smithi184 ceph-mon[100622]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:57 smithi184 ceph-mon[102844]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:04:59.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:01.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:01.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:01.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:01.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[102844]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[100622]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:01 smithi196 ceph-mon[76349]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:02] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-21T00:05:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:03 smithi184 ceph-mon[100622]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:03 smithi184 ceph-mon[102844]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:03 smithi196 ceph-mon[76349]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:05 smithi184 ceph-mon[100622]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:05 smithi184 ceph-mon[102844]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:05 smithi196 ceph-mon[76349]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:07 smithi184 ceph-mon[100622]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:08.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:07 smithi184 ceph-mon[102844]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:07 smithi196 ceph-mon[76349]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:09 smithi184 ceph-mon[102844]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:09 smithi184 ceph-mon[100622]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:09 smithi196 ceph-mon[76349]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:11.226Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:11.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:11.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:11 smithi184 ceph-mon[102844]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:11 smithi184 ceph-mon[100622]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:11 smithi196 ceph-mon[76349]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:12] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-21T00:05:13.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:13 smithi184 ceph-mon[100622]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:13.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:13 smithi184 ceph-mon[102844]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:13 smithi196 ceph-mon[76349]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:14.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:14.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:15 smithi184 ceph-mon[100622]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:15 smithi184 ceph-mon[102844]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:15 smithi196 ceph-mon[76349]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:16 smithi184 ceph-mon[100622]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:16 smithi184 ceph-mon[102844]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:16 smithi196 ceph-mon[76349]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:19 smithi196 ceph-mon[76349]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[100622]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[102844]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:21.226Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:21.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:21.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:21 smithi196 ceph-mon[76349]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[100622]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[102844]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:22] "GET /metrics HTTP/1.1" 200 38935 "" "Prometheus/2.43.0" 2024-09-21T00:05:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:23 smithi196 ceph-mon[76349]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:23 smithi184 ceph-mon[100622]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:23 smithi184 ceph-mon[102844]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:24.309 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:05:24.837 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 4h - - 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 5s ago 4h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 5s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:05:24.838 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 5s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 5s ago 4h 432M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 5s ago 4h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 5s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:05:24.839 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 5s ago 4h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 5s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 5s ago 4h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 5s ago 4h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 4h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 4h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 4h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:05:24.840 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:05:24.841 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 4h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:05:24.841 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 5s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:05:24.841 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:05:24.841 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:05:24.841 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:25.144 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:05:25.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:05:25.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:05:25.145 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:05:25.412 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:05:25.412 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:05:25.412 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:05:25.413 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:05:25.414 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:05:25.414 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:05:25.414 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:05:25.414 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:05:25.723 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:05:25.723 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:25.723 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:25.723 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:25.723 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[100622]: from='client.71171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[100622]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[100622]: from='client.61215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/863710122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[102844]: from='client.71171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.724 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[102844]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:25.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[102844]: from='client.61215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/863710122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:25 smithi196 ceph-mon[76349]: from='client.71171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:25 smithi196 ceph-mon[76349]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:25 smithi196 ceph-mon[76349]: from='client.61215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/863710122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:26.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:26 smithi196 ceph-mon[76349]: from='client.71183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:26.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:26 smithi196 ceph-mon[76349]: from='client.71195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:26.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3037989396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[100622]: from='client.71183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[100622]: from='client.71195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3037989396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[102844]: from='client.71183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[102844]: from='client.71195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3037989396' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:27 smithi196 ceph-mon[76349]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:27 smithi184 ceph-mon[100622]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:27 smithi184 ceph-mon[102844]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:29 smithi196 ceph-mon[76349]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[100622]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[102844]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:31.227Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:31.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:31.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:31.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:31 smithi196 ceph-mon[76349]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:31 smithi184 ceph-mon[100622]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:31 smithi184 ceph-mon[102844]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:05:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:33 smithi184 ceph-mon[100622]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:33 smithi184 ceph-mon[102844]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:33 smithi196 ceph-mon[76349]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:35 smithi196 ceph-mon[76349]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:35 smithi184 ceph-mon[100622]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:35 smithi184 ceph-mon[102844]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:37 smithi196 ceph-mon[76349]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:37 smithi184 ceph-mon[100622]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:37 smithi184 ceph-mon[102844]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:39 smithi196 ceph-mon[76349]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[100622]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[102844]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:41.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:41.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:41.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:41 smithi196 ceph-mon[76349]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:41.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[100622]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[102844]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:05:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:43 smithi196 ceph-mon[76349]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:43 smithi184 ceph-mon[100622]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:43 smithi184 ceph-mon[102844]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:45 smithi184 ceph-mon[100622]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:45 smithi184 ceph-mon[102844]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376439]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376439]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376442]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:46.049 INFO:journalctl@ceph.osd.0.smithi184.stdout:Sep 21 00:05:45 smithi184 sudo[376442]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:45 smithi196 ceph-mon[76349]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:47 smithi184 ceph-mon[100622]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:47 smithi184 ceph-mon[102844]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:47 smithi196 ceph-mon[76349]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:49 smithi184 ceph-mon[100622]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:49 smithi184 ceph-mon[102844]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:49 smithi196 ceph-mon[76349]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:51.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:51.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:05:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:05:51.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:05:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:51 smithi184 ceph-mon[100622]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:51 smithi184 ceph-mon[102844]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:51 smithi196 ceph-mon[76349]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:05:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:05:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:53 smithi196 ceph-mon[76349]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:53 smithi184 ceph-mon[100622]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:54.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:53 smithi184 ceph-mon[102844]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:55.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:55 smithi184 ceph-mon[100622]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:55.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:55 smithi184 ceph-mon[102844]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:56.029 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:05:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:55 smithi196 ceph-mon[76349]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:56.146 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376582]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376546]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376546]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376568]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376568]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376568]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.147 INFO:journalctl@ceph.osd.1.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376568]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280185]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280185]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280185]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280185]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.396 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280188]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280188]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280197]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280197]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280197]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280191]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280191]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280191]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.397 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280191]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.398 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280194]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.398 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280194]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.398 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280194]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.398 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280194]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376582]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376582]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376582]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376590]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376590]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376590]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.2.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376590]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376598]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376598]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.441 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376598]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.442 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376598]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.442 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.442 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.442 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.442 INFO:journalctl@ceph.osd.3.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376611]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 4h - - 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 4h - - 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 17s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 17s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 15s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 17s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 17s ago 4h 433M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 15s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 17s ago 4h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:05:56.669 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 17s ago 4h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 15s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 17s ago 4h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 17s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 17s ago 4h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 17s ago 4h 95.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 15s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 15s ago 4h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 15s ago 4h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 15s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 15s ago 4h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 17s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 17s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:05:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 15s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376698]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376698]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376698]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376698]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376712]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376712]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376712]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376712]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376715]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376715]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376719]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376719]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376719]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:56 smithi184 sudo[376719]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280200]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.890 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280200]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280209]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280209]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280209]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280209]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280212]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280212]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280203]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280203]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280206]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280206]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:05:56.891 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 21 00:05:56 smithi196 sudo[280206]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:05:56.987 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:05:56.987 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:05:56.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:05:56.989 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:05:57.248 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='client.71207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.249 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='client.74590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.250 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1905198445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='client.71207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='client.74590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1905198445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:05:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:05:57.264 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:05:57.264 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:05:57.264 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:05:57.264 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='client.71207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='client.74590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:05:57.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1905198445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:57.573 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:05:57.573 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:57.573 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:57.573 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:57.574 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:58 smithi184 ceph-mon[100622]: from='client.61272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1911842382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:58 smithi184 ceph-mon[102844]: from='client.61272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1911842382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:58 smithi196 ceph-mon[76349]: from='client.61272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1911842382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:59 smithi196 ceph-mon[76349]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:05:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[100622]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[102844]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:05:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:00.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:01.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:01.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:01.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:01.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:01.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[100622]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[102844]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:01 smithi196 ceph-mon[76349]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:06:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:03 smithi184 ceph-mon[100622]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:03 smithi184 ceph-mon[102844]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:03 smithi196 ceph-mon[76349]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:05 smithi196 ceph-mon[76349]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:05 smithi184 ceph-mon[100622]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:05 smithi184 ceph-mon[102844]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:07 smithi196 ceph-mon[76349]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:07 smithi184 ceph-mon[100622]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:07 smithi184 ceph-mon[102844]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:09.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:09 smithi196 ceph-mon[76349]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:09 smithi184 ceph-mon[100622]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:09 smithi184 ceph-mon[102844]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:06:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:11.230Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:11.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:11.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:11 smithi196 ceph-mon[76349]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:11 smithi184 ceph-mon[102844]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:12.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:11 smithi184 ceph-mon[100622]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:06:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:13 smithi196 ceph-mon[76349]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:13 smithi184 ceph-mon[100622]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:13 smithi184 ceph-mon[102844]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:15 smithi196 ceph-mon[76349]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:15 smithi184 ceph-mon[100622]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:15 smithi184 ceph-mon[102844]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:17 smithi196 ceph-mon[76349]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:18.020 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:17 smithi184 ceph-mon[100622]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:18.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:17 smithi184 ceph-mon[102844]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[100622]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[102844]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:19 smithi196 ceph-mon[76349]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:21.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:21.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:21.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:21 smithi196 ceph-mon[76349]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[102844]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[100622]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:06:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:23 smithi196 ceph-mon[76349]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:23 smithi184 ceph-mon[100622]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:23 smithi184 ceph-mon[102844]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:25 smithi184 ceph-mon[100622]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:25 smithi184 ceph-mon[102844]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:25 smithi196 ceph-mon[76349]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:27.870 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:06:27.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:27 smithi196 ceph-mon[76349]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:27.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:27 smithi184 ceph-mon[100622]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:27.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:27 smithi184 ceph-mon[102844]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:28.404 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 4h - - 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 4h - - 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 8s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 7s ago 4h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 8s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 8s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 8s ago 4h 433M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 8s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 7s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:06:28.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 8s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 8s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 8s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 8s ago 4h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 7s ago 4h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 8s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 8s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:06:28.406 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:06:28.720 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:06:28.721 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:06:28.989 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:06:28.989 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:06:28.989 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:06:28.989 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:06:28.990 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:06:29.305 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:06:29.305 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:29.305 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:29.305 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:29.305 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='client.61284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='client.61290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='client.61296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/557406023' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='client.61284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='client.61290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='client.61296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/557406023' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='client.61284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='client.61290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='client.61296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/557406023' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:30 smithi196 ceph-mon[76349]: from='client.61308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2553060316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:30 smithi184 ceph-mon[100622]: from='client.61308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2553060316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:30 smithi184 ceph-mon[102844]: from='client.61308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2553060316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:31 smithi184 ceph-mon[102844]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:31 smithi184 ceph-mon[100622]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:31.232Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:31.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:31.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:31 smithi196 ceph-mon[76349]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:06:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:33 smithi184 ceph-mon[100622]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:33 smithi184 ceph-mon[102844]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:33 smithi196 ceph-mon[76349]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:35 smithi196 ceph-mon[76349]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:35 smithi184 ceph-mon[100622]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:35 smithi184 ceph-mon[102844]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:37 smithi196 ceph-mon[76349]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:37.895 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:37 smithi184 ceph-mon[100622]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:37.896 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:37 smithi184 ceph-mon[102844]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:39 smithi196 ceph-mon[76349]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:39 smithi184 ceph-mon[100622]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:39 smithi184 ceph-mon[102844]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[100622]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:40 smithi184 ceph-mon[102844]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:40 smithi196 ceph-mon[76349]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:41.233Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:41.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:41.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:06:42.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:06:43.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:43 smithi196 ceph-mon[76349]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:43 smithi184 ceph-mon[100622]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:43 smithi184 ceph-mon[102844]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:44.511 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:45 smithi196 ceph-mon[76349]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:45 smithi184 ceph-mon[100622]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:45 smithi184 ceph-mon[102844]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:47.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:47 smithi196 ceph-mon[76349]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:47 smithi184 ceph-mon[100622]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:47 smithi184 ceph-mon[102844]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:49 smithi196 ceph-mon[76349]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:49 smithi184 ceph-mon[100622]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:49 smithi184 ceph-mon[102844]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:51.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:51.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:06:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:06:51.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:06:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:51 smithi196 ceph-mon[76349]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:51 smithi184 ceph-mon[100622]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:51 smithi184 ceph-mon[102844]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:06:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:06:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:53 smithi196 ceph-mon[76349]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:53 smithi184 ceph-mon[100622]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:53 smithi184 ceph-mon[102844]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:55 smithi184 ceph-mon[100622]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:55 smithi184 ceph-mon[102844]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:55 smithi196 ceph-mon[76349]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:57 smithi184 ceph-mon[100622]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:57 smithi184 ceph-mon[102844]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:57.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:57 smithi196 ceph-mon[76349]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:59.591 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[100622]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[102844]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:59 smithi196 ceph-mon[76349]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 4h - - 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 18s ago 4h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 18s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 4h 434M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:07:00.145 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 18s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 4h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 18s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 18s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 18s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 18s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 18s ago 4h 75.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:07:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:07:00.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:07:00.147 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 18s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:07:00.469 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:07:00.469 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:07:00.469 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:07:00.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:00.469 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:07:00.470 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:07:00.751 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:07:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:07:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:07:00.751 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:07:00.752 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[100622]: from='client.61320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4195605761' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[102844]: from='client.61320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:00.754 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4195605761' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:01.096 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:07:01.096 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:01.096 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:01.096 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:01.096 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:00 smithi196 ceph-mon[76349]: from='client.61320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:01.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4195605761' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:01.235Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:01.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:01.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='client.71288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='client.61332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/163339235' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='client.71288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='client.61332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/163339235' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='client.71288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='client.61332 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/163339235' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:02] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:07:02.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:02 smithi196 ceph-mon[76349]: from='client.71306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:02 smithi184 ceph-mon[100622]: from='client.71306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:02 smithi184 ceph-mon[102844]: from='client.71306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:03.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:03 smithi196 ceph-mon[76349]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:03 smithi184 ceph-mon[100622]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:03 smithi184 ceph-mon[102844]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:05 smithi184 ceph-mon[100622]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:05 smithi184 ceph-mon[102844]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:05 smithi196 ceph-mon[76349]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:07 smithi184 ceph-mon[100622]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:07 smithi184 ceph-mon[102844]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:07 smithi196 ceph-mon[76349]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:09 smithi184 ceph-mon[100622]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:09 smithi184 ceph-mon[102844]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:09 smithi196 ceph-mon[76349]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:11.235Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:11.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:11.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:11 smithi184 ceph-mon[100622]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:11 smithi184 ceph-mon[102844]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:11 smithi196 ceph-mon[76349]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:12] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:07:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:13 smithi184 ceph-mon[100622]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:13 smithi184 ceph-mon[102844]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:13 smithi196 ceph-mon[76349]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:15 smithi184 ceph-mon[100622]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:15 smithi184 ceph-mon[102844]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:15 smithi196 ceph-mon[76349]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:17 smithi184 ceph-mon[100622]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:17 smithi184 ceph-mon[102844]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:17 smithi196 ceph-mon[76349]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:19.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:19.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:19 smithi184 ceph-mon[100622]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:19.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:19.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:19 smithi184 ceph-mon[102844]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:19 smithi196 ceph-mon[76349]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[102844]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-mon[100622]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:21.236Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:21.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:21.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:21.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:21 smithi196 ceph-mon[76349]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:07:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:23 smithi196 ceph-mon[76349]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:23 smithi184 ceph-mon[100622]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:23 smithi184 ceph-mon[102844]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:25 smithi196 ceph-mon[76349]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:25 smithi184 ceph-mon[100622]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:25 smithi184 ceph-mon[102844]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:27.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:27 smithi196 ceph-mon[76349]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:27 smithi184 ceph-mon[100622]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:27 smithi184 ceph-mon[102844]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[100622]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[102844]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:29 smithi196 ceph-mon[76349]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:07:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:31.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:31.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:31.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:31.919 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:07:31.919 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-21T00:07:31.919 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 4h 435M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:07:31.920 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:07:31.921 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 4h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:07:31.922 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 4h 75.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:07:31.923 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:07:31.923 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:07:31.923 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:07:31.923 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:07:31.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:31 smithi184 ceph-mon[100622]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:31.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:31 smithi184 ceph-mon[102844]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:31 smithi196 ceph-mon[76349]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:32.229 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:07:32.229 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:07:32.229 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:07:32.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:07:32.231 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:07:32.232 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:07:32.512 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:07:32.513 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:07:32.514 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:07:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[100622]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[100622]: from='client.61362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1395952603' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[102844]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[102844]: from='client.61362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1395952603' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:32.836 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:07:32.836 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:32.836 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:32.836 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:32.837 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:32 smithi196 ceph-mon[76349]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:32 smithi196 ceph-mon[76349]: from='client.61362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1395952603' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[102844]: from='client.71330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[102844]: from='client.71342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[102844]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1910126626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[100622]: from='client.71330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[100622]: from='client.71342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[100622]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:33.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1910126626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:33 smithi196 ceph-mon[76349]: from='client.71330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:33 smithi196 ceph-mon[76349]: from='client.71342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:33 smithi196 ceph-mon[76349]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1910126626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:35 smithi184 ceph-mon[100622]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:35 smithi184 ceph-mon[102844]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:35 smithi196 ceph-mon[76349]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:37 smithi184 ceph-mon[100622]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:37 smithi184 ceph-mon[102844]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:37 smithi196 ceph-mon[76349]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:39 smithi184 ceph-mon[100622]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:39 smithi184 ceph-mon[102844]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:39 smithi196 ceph-mon[76349]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:41 smithi196 ceph-mon[76349]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[102844]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:41.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-mon[100622]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:41.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:41.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:41.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:41.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:07:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:07:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:43 smithi184 ceph-mon[100622]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:43 smithi184 ceph-mon[102844]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:43.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:43 smithi196 ceph-mon[76349]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:44.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:44.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:44 smithi196 ceph-mon[76349]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:44 smithi184 ceph-mon[100622]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:44 smithi184 ceph-mon[102844]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:47.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:47 smithi196 ceph-mon[76349]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:47 smithi184 ceph-mon[100622]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:47 smithi184 ceph-mon[102844]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:49 smithi184 ceph-mon[102844]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:49 smithi184 ceph-mon[100622]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:49 smithi196 ceph-mon[76349]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:51.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:51.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:07:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:07:51.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:07:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:51 smithi184 ceph-mon[100622]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:51 smithi184 ceph-mon[102844]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:51 smithi196 ceph-mon[76349]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:07:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:07:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:53 smithi184 ceph-mon[100622]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:53 smithi184 ceph-mon[102844]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:53 smithi196 ceph-mon[76349]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:55 smithi184 ceph-mon[100622]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:55 smithi184 ceph-mon[102844]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:55 smithi196 ceph-mon[76349]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:57 smithi196 ceph-mon[76349]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:57 smithi184 ceph-mon[100622]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:57 smithi184 ceph-mon[102844]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:00.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:00.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[100622]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:00.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:00.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:00.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[102844]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:00.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:59 smithi196 ceph-mon[76349]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:01.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[100622]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-mon[102844]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:01.524 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:01.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:01.524 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:01.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:01.524 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:01.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:01 smithi196 ceph-mon[76349]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:08:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[100622]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:02 smithi184 ceph-mon[102844]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:03.132 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:08:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:02 smithi196 ceph-mon[76349]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:03.687 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 4h - - 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 1s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 1s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 4h 436M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 1s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 1s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:08:03.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 1s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 1s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 1s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 1s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 1s ago 4h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:08:03.689 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 1s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:08:04.003 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:08:04.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:08:04.004 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:08:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:03 smithi196 ceph-mon[76349]: from='client.61392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:03 smithi196 ceph-mon[76349]: from='client.61398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:03 smithi196 ceph-mon[76349]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:08:04.271 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:08:04.271 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:08:04.272 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:08:04.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[102844]: from='client.61392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[102844]: from='client.61398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[102844]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[100622]: from='client.61392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[100622]: from='client.61398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:03 smithi184 ceph-mon[100622]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:04.583 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:08:04.583 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:04.583 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:04.583 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:04.583 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:05.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2243151187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:05.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:04 smithi196 ceph-mon[76349]: from='client.61416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:05.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4184050588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:05.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:04 smithi196 ceph-mon[76349]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2243151187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[100622]: from='client.61416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4184050588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[100622]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2243151187' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[102844]: from='client.61416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4184050588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:04 smithi184 ceph-mon[102844]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:07 smithi184 ceph-mon[100622]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:07 smithi184 ceph-mon[102844]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:07 smithi196 ceph-mon[76349]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:09 smithi184 ceph-mon[100622]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:09 smithi184 ceph-mon[102844]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:09 smithi196 ceph-mon[76349]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:11.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:11.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:11.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:11 smithi184 ceph-mon[100622]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:11 smithi184 ceph-mon[102844]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:11 smithi196 ceph-mon[76349]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:08:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:13 smithi184 ceph-mon[100622]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:13 smithi184 ceph-mon[102844]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:13 smithi196 ceph-mon[76349]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:08:13.812271111Z level=info msg="Completed cleanup jobs" duration=12.812371ms 2024-09-21T00:08:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:15 smithi184 ceph-mon[100622]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:15 smithi184 ceph-mon[102844]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:15 smithi196 ceph-mon[76349]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:17 smithi184 ceph-mon[100622]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:17 smithi184 ceph-mon[102844]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:17 smithi196 ceph-mon[76349]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:19.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:19.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:19 smithi184 ceph-mon[100622]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:19.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:19.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:19 smithi184 ceph-mon[102844]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:19 smithi196 ceph-mon[76349]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:21.416 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:21.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:21.416 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:21.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:21.416 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:21.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:21.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.793 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[100622]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:21.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:21 smithi184 ceph-mon[102844]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:21 smithi196 ceph-mon[76349]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[100622]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:22 smithi184 ceph-mon[102844]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:22 smithi196 ceph-mon[76349]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:25 smithi196 ceph-mon[76349]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:25 smithi184 ceph-mon[100622]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:25 smithi184 ceph-mon[102844]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:27 smithi184 ceph-mon[100622]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:28.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:27 smithi184 ceph-mon[102844]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:28.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:27 smithi196 ceph-mon[76349]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[100622]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[102844]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:29 smithi196 ceph-mon[76349]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:31.552 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:31.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:31.552 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:31.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:31.552 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:31.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:31 smithi184 ceph-mon[102844]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:32.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:31 smithi184 ceph-mon[100622]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:31 smithi196 ceph-mon[76349]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:08:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:33 smithi184 ceph-mon[100622]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:33 smithi184 ceph-mon[102844]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:33 smithi196 ceph-mon[76349]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:34.879 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:08:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:08:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-21T00:08:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 4h - - 2024-09-21T00:08:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 13s ago 4h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 13s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 4h 436M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 13s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:08:35.403 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 13s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:08:35.404 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 13s ago 4h 76.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:08:35.405 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:08:35.712 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:08:35.712 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:08:35.712 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:08:35.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:35.712 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:08:35.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:08:35.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:08:35.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:08:35.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:35 smithi184 ceph-mon[100622]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:35.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:35 smithi184 ceph-mon[102844]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:35.985 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:08:35.985 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:08:35.985 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:08:35.985 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:08:35.985 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:08:35.986 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:08:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:35 smithi196 ceph-mon[76349]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:36.295 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:08:36.295 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:36.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:36.295 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:36.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[100622]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[100622]: from='client.71396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[100622]: from='client.71402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3928844051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:37.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2819217905' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:37.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[102844]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[102844]: from='client.71396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[102844]: from='client.71402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3928844051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:37.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2819217905' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:36 smithi196 ceph-mon[76349]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:36 smithi196 ceph-mon[76349]: from='client.71396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:36 smithi196 ceph-mon[76349]: from='client.71402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3928844051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2819217905' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:37 smithi184 ceph-mon[100622]: from='client.61452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:37 smithi184 ceph-mon[100622]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:37 smithi184 ceph-mon[102844]: from='client.61452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:37 smithi184 ceph-mon[102844]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:37 smithi196 ceph-mon[76349]: from='client.61452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:37 smithi196 ceph-mon[76349]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:39 smithi184 ceph-mon[100622]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:39 smithi184 ceph-mon[102844]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:39 smithi196 ceph-mon[76349]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:41.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:41.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:41.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:41.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:41.509 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:41.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:41.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[102844]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:41.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:41 smithi184 ceph-mon[100622]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:41 smithi196 ceph-mon[76349]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:08:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[100622]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:43 smithi184 ceph-mon[102844]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:08:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:43 smithi196 ceph-mon[76349]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:45 smithi196 ceph-mon[76349]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:45 smithi184 ceph-mon[100622]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:45 smithi184 ceph-mon[102844]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:47 smithi196 ceph-mon[76349]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:47 smithi184 ceph-mon[100622]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:47 smithi184 ceph-mon[102844]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:49 smithi184 ceph-mon[100622]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:49 smithi184 ceph-mon[102844]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:49 smithi196 ceph-mon[76349]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:51.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:51.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:08:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:08:51.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:08:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:51 smithi184 ceph-mon[100622]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:51 smithi184 ceph-mon[102844]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:51 smithi196 ceph-mon[76349]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:08:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:53 smithi184 ceph-mon[100622]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:53 smithi184 ceph-mon[102844]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:53 smithi196 ceph-mon[76349]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:55 smithi184 ceph-mon[100622]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:55 smithi184 ceph-mon[102844]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:55 smithi196 ceph-mon[76349]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:57.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:57 smithi196 ceph-mon[76349]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:57 smithi184 ceph-mon[100622]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:57 smithi184 ceph-mon[102844]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[100622]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[102844]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:59 smithi196 ceph-mon[76349]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:01.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:01.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:01.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:01.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[100622]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[102844]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:01 smithi196 ceph-mon[76349]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:09:03.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.504 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[100622]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:03.504 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:03 smithi184 ceph-mon[102844]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:03 smithi196 ceph-mon[76349]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:05 smithi184 ceph-mon[100622]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:05 smithi184 ceph-mon[102844]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:05 smithi196 ceph-mon[76349]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:06.589 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 4h 438M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:09:07.114 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 4h 76.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:09:07.115 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:09:07.423 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:09:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:09:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:09:07.425 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:09:07.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[100622]: from='client.61464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:07.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[100622]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:07.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2954901928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:09:07.694 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:09:07.695 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:09:07.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:07 smithi196 ceph-mon[76349]: from='client.61464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:07.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:07 smithi196 ceph-mon[76349]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:07.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2954901928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:08.007 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:09:08.007 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:08.007 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:08.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[102844]: from='client.61464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[102844]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:08.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2954901928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[100622]: from='client.61470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[100622]: from='client.71438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[100622]: from='client.71450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2087429629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[102844]: from='client.61470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[102844]: from='client.71438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[102844]: from='client.71450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2087429629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:09.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:08 smithi196 ceph-mon[76349]: from='client.61470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:09.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:08 smithi196 ceph-mon[76349]: from='client.71438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:09.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:08 smithi196 ceph-mon[76349]: from='client.71450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:09.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2087429629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:09 smithi184 ceph-mon[100622]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:09 smithi184 ceph-mon[102844]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:09 smithi196 ceph-mon[76349]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:11.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:11.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:11.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:11 smithi184 ceph-mon[102844]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:11 smithi184 ceph-mon[100622]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:11 smithi196 ceph-mon[76349]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:09:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:13 smithi184 ceph-mon[100622]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:13 smithi184 ceph-mon[102844]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:13 smithi196 ceph-mon[76349]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:15 smithi184 ceph-mon[100622]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:15 smithi184 ceph-mon[102844]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:15 smithi196 ceph-mon[76349]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:17 smithi184 ceph-mon[100622]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:17 smithi184 ceph-mon[102844]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:17 smithi196 ceph-mon[76349]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:19 smithi184 ceph-mon[100622]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:19 smithi184 ceph-mon[102844]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:19 smithi196 ceph-mon[76349]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:21.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:21.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:21.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[100622]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[102844]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:21.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:21.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:21 smithi196 ceph-mon[76349]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:09:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:23 smithi196 ceph-mon[76349]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[102844]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:23 smithi184 ceph-mon[100622]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:25 smithi184 ceph-mon[100622]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:25 smithi184 ceph-mon[102844]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:25 smithi196 ceph-mon[76349]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:27 smithi184 ceph-mon[100622]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:27 smithi184 ceph-mon[102844]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:27 smithi196 ceph-mon[76349]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[100622]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[102844]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:29 smithi196 ceph-mon[76349]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:31.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:31.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:31.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:32.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:31 smithi184 ceph-mon[100622]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:31 smithi184 ceph-mon[102844]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:31 smithi196 ceph-mon[76349]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:09:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:33 smithi184 ceph-mon[100622]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:33 smithi184 ceph-mon[102844]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:33 smithi196 ceph-mon[76349]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:35 smithi184 ceph-mon[100622]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:35 smithi184 ceph-mon[102844]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:35 smithi196 ceph-mon[76349]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:37 smithi184 ceph-mon[100622]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:37 smithi184 ceph-mon[102844]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:37 smithi196 ceph-mon[76349]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:38.305 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:09:38.835 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 4h 438M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 4h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:09:38.836 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:09:38.837 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:09:38.837 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:09:38.837 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:09:38.837 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:09:38.837 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:09:39.142 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:09:39.142 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:09:39.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:39.143 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:09:39.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:09:39.411 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:09:39.412 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:09:39.413 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:09:39.413 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:09:39.413 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:09:39.413 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:09:39.413 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:09:39.721 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:09:39.721 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:39.721 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:39.721 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:39.721 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[100622]: from='client.71462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[100622]: from='client.71468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[100622]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/874995495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[102844]: from='client.71462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[102844]: from='client.71468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[102844]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/874995495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:39 smithi196 ceph-mon[76349]: from='client.71462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:39 smithi196 ceph-mon[76349]: from='client.71468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:39 smithi196 ceph-mon[76349]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/874995495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:40.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[100622]: from='client.61512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.956 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[100622]: from='client.61524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2934324985' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:40.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[102844]: from='client.61512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[102844]: from='client.61524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:40.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2934324985' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:40 smithi196 ceph-mon[76349]: from='client.61512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:40 smithi196 ceph-mon[76349]: from='client.61524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2934324985' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:41.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:41.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:41.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[100622]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[102844]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:41 smithi196 ceph-mon[76349]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[100622]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:43 smithi184 ceph-mon[102844]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:09:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:43 smithi196 ceph-mon[76349]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:45 smithi184 ceph-mon[100622]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:45 smithi184 ceph-mon[102844]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:45 smithi196 ceph-mon[76349]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:47 smithi184 ceph-mon[100622]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:47 smithi184 ceph-mon[102844]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:48.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:47 smithi196 ceph-mon[76349]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:49.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:49.776 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:49 smithi196 ceph-mon[76349]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:49 smithi184 ceph-mon[100622]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:49 smithi184 ceph-mon[102844]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:51.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:51.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:09:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:09:51.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:09:51.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:51 smithi196 ceph-mon[76349]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:51 smithi184 ceph-mon[100622]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:51 smithi184 ceph-mon[102844]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:09:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:09:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:53 smithi184 ceph-mon[100622]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:53 smithi184 ceph-mon[102844]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:53 smithi196 ceph-mon[76349]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:55 smithi184 ceph-mon[100622]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:55 smithi184 ceph-mon[102844]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:55 smithi196 ceph-mon[76349]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:57.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:57 smithi196 ceph-mon[76349]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:57 smithi184 ceph-mon[100622]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:57 smithi184 ceph-mon[102844]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[100622]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[102844]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:59 smithi196 ceph-mon[76349]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:01.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:01.247Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:01.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:01.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[100622]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[102844]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:01 smithi196 ceph-mon[76349]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:02] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[102844]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:03 smithi184 ceph-mon[100622]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:03 smithi196 ceph-mon[76349]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:05 smithi184 ceph-mon[100622]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:05 smithi184 ceph-mon[102844]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:05 smithi196 ceph-mon[76349]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:07.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:07 smithi196 ceph-mon[76349]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:07 smithi184 ceph-mon[100622]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:07 smithi184 ceph-mon[102844]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:09.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:09.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:09 smithi196 ceph-mon[76349]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:10.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:09 smithi184 ceph-mon[100622]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:10.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:09 smithi184 ceph-mon[102844]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.017 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:10:10.542 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:10:10.542 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 4h - - 2024-09-21T00:10:10.542 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 4h - - 2024-09-21T00:10:10.542 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 9s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:10:10.542 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 7s ago 4h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 9s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 9s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 4h 440M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 9s ago 4h 18.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 7s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 9s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 9s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 9s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 9s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:10:10.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 7s ago 4h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 9s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 9s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:10:10.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:10.857 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:10:10.858 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:10:11.127 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:10:11.127 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:10:11.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:10:11.437 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:11.437 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:11.437 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:11.437 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:11.437 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:11.437 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:11.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:11.438 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:11.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:11.438 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:11.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: from='client.71498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: from='client.71504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: from='client.71510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1045647915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/450339124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: from='client.71498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: from='client.71504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: from='client.71510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1045647915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/450339124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: from='client.71498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: from='client.71504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: from='client.71510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1045647915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/450339124' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:12] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:10:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:12 smithi184 ceph-mon[100622]: from='client.61560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:12 smithi184 ceph-mon[102844]: from='client.61560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:12 smithi196 ceph-mon[76349]: from='client.61560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:13 smithi184 ceph-mon[100622]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:13 smithi184 ceph-mon[102844]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:13 smithi196 ceph-mon[76349]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:15 smithi184 ceph-mon[100622]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:15 smithi184 ceph-mon[102844]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:15 smithi196 ceph-mon[76349]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:17 smithi184 ceph-mon[100622]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:17 smithi184 ceph-mon[102844]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:17 smithi196 ceph-mon[76349]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:18 smithi184 ceph-mon[100622]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:18 smithi184 ceph-mon[102844]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:18 smithi196 ceph-mon[76349]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:21.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:21.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:21.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[100622]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[102844]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:21 smithi196 ceph-mon[76349]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:10:23.973 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[100622]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[102844]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:23.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:23 smithi196 ceph-mon[76349]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:25 smithi184 ceph-mon[100622]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:25 smithi184 ceph-mon[102844]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:25 smithi196 ceph-mon[76349]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:27.977 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:27 smithi184 ceph-mon[100622]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:27.977 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:27 smithi184 ceph-mon[102844]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:27 smithi196 ceph-mon[76349]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:29.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:29.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:29 smithi196 ceph-mon[76349]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[100622]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[102844]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:29.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:31.249Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:31.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:31.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:31 smithi184 ceph-mon[102844]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:31 smithi184 ceph-mon[100622]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:32.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:31 smithi196 ceph-mon[76349]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:10:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:33 smithi184 ceph-mon[100622]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:33 smithi184 ceph-mon[102844]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:33 smithi196 ceph-mon[76349]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:35 smithi184 ceph-mon[100622]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:35 smithi184 ceph-mon[102844]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:35 smithi196 ceph-mon[76349]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:37 smithi184 ceph-mon[100622]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:37.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:37 smithi184 ceph-mon[102844]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:37 smithi196 ceph-mon[76349]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:39 smithi184 ceph-mon[100622]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:39 smithi184 ceph-mon[102844]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:39 smithi196 ceph-mon[76349]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:41.249Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:41.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:41.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:41.722 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[100622]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[102844]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:41 smithi196 ceph-mon[76349]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 4h 441M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:10:42.252 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 76.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:10:42.253 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:10:42.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:10:42.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:10:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:10:42.562 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:10:42.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:10:42.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:10:42.564 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:10:42.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:10:42.564 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:10:42.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:10:42.834 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:42 smithi184 ceph-mon[100622]: from='client.71534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:42.834 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1068547364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:42.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:42 smithi184 ceph-mon[102844]: from='client.71534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:42.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1068547364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:10:42.835 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:10:42.836 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:10:42.836 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:10:42.836 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:10:42.836 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:10:42.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:42 smithi196 ceph-mon[76349]: from='client.71534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:42.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1068547364' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:43.153 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:43.153 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:43.153 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:43.153 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:43.153 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='client.61578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='client.71546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2232715512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='client.61578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='client.71546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2232715512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='client.61578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='client.71546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:10:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2232715512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[100622]: from='client.61596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[100622]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[102844]: from='client.61596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:45.025 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:44 smithi184 ceph-mon[102844]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:44 smithi196 ceph-mon[76349]: from='client.61596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:44 smithi196 ceph-mon[76349]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:47 smithi184 ceph-mon[100622]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:48.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:47 smithi184 ceph-mon[102844]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:47 smithi196 ceph-mon[76349]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:49.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:49.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:49 smithi196 ceph-mon[76349]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:49 smithi184 ceph-mon[100622]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:49 smithi184 ceph-mon[102844]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:51.250Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:51.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:10:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:10:51.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:10:51.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:51 smithi196 ceph-mon[76349]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:51 smithi184 ceph-mon[100622]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:51 smithi184 ceph-mon[102844]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:10:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:10:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:53 smithi184 ceph-mon[100622]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:53 smithi184 ceph-mon[102844]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:53 smithi196 ceph-mon[76349]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:55 smithi184 ceph-mon[100622]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:55 smithi184 ceph-mon[102844]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:55 smithi196 ceph-mon[76349]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:57 smithi184 ceph-mon[100622]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:57 smithi184 ceph-mon[102844]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:57 smithi196 ceph-mon[76349]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[100622]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[102844]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:10:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:59 smithi196 ceph-mon[76349]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:10:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:00 smithi184 ceph-mon[100622]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:00 smithi184 ceph-mon[102844]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:00 smithi196 ceph-mon[76349]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:01.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:01.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:01.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:11:02.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[100622]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[102844]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:03 smithi196 ceph-mon[76349]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:05 smithi184 ceph-mon[100622]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:05 smithi184 ceph-mon[102844]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:05 smithi196 ceph-mon[76349]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:07.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:07 smithi184 ceph-mon[100622]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:07.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:07 smithi184 ceph-mon[102844]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:07 smithi196 ceph-mon[76349]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:09 smithi184 ceph-mon[100622]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:09 smithi184 ceph-mon[102844]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:09 smithi196 ceph-mon[76349]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:11.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:11.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:11.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:11.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:11 smithi196 ceph-mon[76349]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:11 smithi184 ceph-mon[100622]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:11 smithi184 ceph-mon[102844]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:11:13.449 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:11:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:13 smithi196 ceph-mon[76349]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-21T00:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-21T00:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 12s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 12s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 12s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 12s ago 4h 440M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 4h 310M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 12s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 12s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 12s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 12s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 12s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 4h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 12s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 12s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:11:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:13 smithi184 ceph-mon[100622]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:13 smithi184 ceph-mon[102844]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:11:14.863 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:11:14.863 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:14.864 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:14.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[100622]: from='client.71570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:14.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[100622]: from='client.61614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:14.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:14.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/773927114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:14.864 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[102844]: from='client.71570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:14.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[102844]: from='client.61614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:14.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:14.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/773927114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:14 smithi196 ceph-mon[76349]: from='client.71570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:14 smithi196 ceph-mon[76349]: from='client.61614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/773927114' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[102844]: from='client.61620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[102844]: from='client.61632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[102844]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1382578853' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[100622]: from='client.61620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[100622]: from='client.61632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[100622]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1382578853' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:15 smithi196 ceph-mon[76349]: from='client.61620 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:15 smithi196 ceph-mon[76349]: from='client.61632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:15 smithi196 ceph-mon[76349]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1382578853' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:16 smithi184 ceph-mon[102844]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:16 smithi184 ceph-mon[100622]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:16 smithi196 ceph-mon[76349]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:19 smithi184 ceph-mon[102844]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:19 smithi184 ceph-mon[100622]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:19 smithi196 ceph-mon[76349]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:21.253Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:21.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:21.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[100622]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[102844]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:21 smithi196 ceph-mon[76349]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:11:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:23 smithi196 ceph-mon[76349]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[102844]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[100622]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:25 smithi184 ceph-mon[102844]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:25 smithi184 ceph-mon[100622]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:25 smithi196 ceph-mon[76349]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:27 smithi184 ceph-mon[100622]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:27 smithi184 ceph-mon[102844]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:27 smithi196 ceph-mon[76349]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:28 smithi184 ceph-mon[100622]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:28 smithi184 ceph-mon[102844]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:28 smithi196 ceph-mon[76349]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:30.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:30 smithi184 ceph-mon[100622]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:30 smithi184 ceph-mon[102844]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:30 smithi196 ceph-mon[76349]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:31.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:31.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:31.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:11:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:33 smithi184 ceph-mon[100622]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:33 smithi184 ceph-mon[102844]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:33 smithi196 ceph-mon[76349]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:35 smithi184 ceph-mon[100622]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:35 smithi184 ceph-mon[102844]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:35 smithi196 ceph-mon[76349]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:37 smithi184 ceph-mon[100622]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:37 smithi184 ceph-mon[102844]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:37 smithi196 ceph-mon[76349]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:39 smithi196 ceph-mon[76349]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:39 smithi184 ceph-mon[100622]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:39 smithi184 ceph-mon[102844]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:40 smithi184 ceph-mon[100622]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:40 smithi184 ceph-mon[102844]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:40 smithi196 ceph-mon[76349]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:41.255Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:41.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:41.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:42.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[102844]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[100622]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:43.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:43 smithi196 ceph-mon[76349]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:11:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:45.159 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:11:45.685 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:11:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 4h 442M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:11:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 70.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:11:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:11:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:11:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:11:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:11:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:11:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:11:46.000 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:11:46.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:45 smithi184 ceph-mon[102844]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:46.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:45 smithi184 ceph-mon[100622]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:45 smithi196 ceph-mon[76349]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:11:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:11:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:11:46.278 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:11:46.588 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:11:46.588 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:46.588 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:46.588 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:46.588 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:46.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[102844]: from='client.71606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[102844]: from='client.61650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[102844]: from='client.61656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2613803804' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:46.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1457885172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:46.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[100622]: from='client.71606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[100622]: from='client.61650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[100622]: from='client.61656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:46.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2613803804' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:46.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1457885172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:46 smithi196 ceph-mon[76349]: from='client.71606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:46 smithi196 ceph-mon[76349]: from='client.61650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:46 smithi196 ceph-mon[76349]: from='client.61656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2613803804' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1457885172' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:47.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:47 smithi184 ceph-mon[100622]: from='client.71630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:47.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:47 smithi184 ceph-mon[100622]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:47.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:47 smithi184 ceph-mon[102844]: from='client.71630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:47.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:47 smithi184 ceph-mon[102844]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:47 smithi196 ceph-mon[76349]: from='client.71630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:47 smithi196 ceph-mon[76349]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:48.808 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:48 smithi184 ceph-mon[102844]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:48.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:48.809 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:48 smithi184 ceph-mon[100622]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:48 smithi196 ceph-mon[76349]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:51.255Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:51.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:11:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:11:51.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:11:52.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:51 smithi196 ceph-mon[76349]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:51 smithi184 ceph-mon[102844]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:51 smithi184 ceph-mon[100622]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:11:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:11:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:53 smithi184 ceph-mon[100622]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:53 smithi184 ceph-mon[102844]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:54.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:53 smithi196 ceph-mon[76349]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:54 smithi196 ceph-mon[76349]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:54 smithi184 ceph-mon[100622]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:54 smithi184 ceph-mon[102844]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:57 smithi184 ceph-mon[100622]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:57 smithi184 ceph-mon[102844]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:57 smithi196 ceph-mon[76349]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[100622]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[102844]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:59 smithi196 ceph-mon[76349]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:00 smithi184 ceph-mon[102844]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:00 smithi184 ceph-mon[100622]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:00 smithi196 ceph-mon[76349]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:01.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:01.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:01.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:12:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[100622]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:02 smithi184 ceph-mon[102844]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:02 smithi196 ceph-mon[76349]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:05 smithi196 ceph-mon[76349]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:05 smithi184 ceph-mon[100622]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:05 smithi184 ceph-mon[102844]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:08.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:07 smithi184 ceph-mon[100622]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:08.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:07 smithi184 ceph-mon[102844]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:07 smithi196 ceph-mon[76349]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:09.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:09.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:08 smithi184 ceph-mon[100622]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:09.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:09.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:08 smithi184 ceph-mon[102844]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:08 smithi196 ceph-mon[76349]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:11.257Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:11.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:11.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:11 smithi184 ceph-mon[100622]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:11 smithi184 ceph-mon[102844]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:11 smithi196 ceph-mon[76349]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:12:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:12 smithi184 ceph-mon[102844]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:12 smithi184 ceph-mon[100622]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:13.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:12 smithi196 ceph-mon[76349]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:15 smithi196 ceph-mon[76349]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:15 smithi184 ceph-mon[100622]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:15 smithi184 ceph-mon[102844]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:16.885 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:12:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:16 smithi184 ceph-mon[100622]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:16 smithi184 ceph-mon[102844]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:16 smithi196 ceph-mon[76349]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 4h - - 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:12:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 15s ago 4h 442M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 15s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 15s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 15s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 15s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 14s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 14s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:12:17.413 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:12:17.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:12:17.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:12:17.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 15s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:12:17.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 14s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:12:17.721 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:12:17.721 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:12:17.721 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:12:17.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:12:17.722 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:12:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:12:17.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[102844]: from='client.71642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[102844]: from='client.71648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[102844]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2678456591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[100622]: from='client.71642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[100622]: from='client.71648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[100622]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:17.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2678456591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:17 smithi196 ceph-mon[76349]: from='client.71642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:17 smithi196 ceph-mon[76349]: from='client.71648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:17 smithi196 ceph-mon[76349]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2678456591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:18.308 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:12:18.308 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:18.308 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:18.308 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:18.308 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[100622]: from='client.71666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3455538690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[100622]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[102844]: from='client.71666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3455538690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:18 smithi184 ceph-mon[102844]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:18 smithi196 ceph-mon[76349]: from='client.71666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3455538690' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:18 smithi196 ceph-mon[76349]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:20 smithi184 ceph-mon[100622]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:20 smithi184 ceph-mon[102844]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:20 smithi196 ceph-mon[76349]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:21.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:21.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:21.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[102844]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:22 smithi184 ceph-mon[100622]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:23.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:22 smithi196 ceph-mon[76349]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:25 smithi184 ceph-mon[100622]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:25 smithi184 ceph-mon[102844]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:25 smithi196 ceph-mon[76349]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:26 smithi184 ceph-mon[100622]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:26 smithi184 ceph-mon[102844]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:26 smithi196 ceph-mon[76349]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:28 smithi184 ceph-mon[102844]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:28 smithi184 ceph-mon[100622]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:28 smithi196 ceph-mon[76349]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:30.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:30 smithi184 ceph-mon[100622]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:30 smithi184 ceph-mon[102844]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:30 smithi196 ceph-mon[76349]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:31.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:31.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:31.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:12:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:32 smithi184 ceph-mon[100622]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:32 smithi184 ceph-mon[102844]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:33.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:32 smithi196 ceph-mon[76349]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:34 smithi184 ceph-mon[100622]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:34 smithi184 ceph-mon[102844]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:35.122 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:34 smithi196 ceph-mon[76349]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:36 smithi184 ceph-mon[100622]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:36 smithi184 ceph-mon[102844]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:36 smithi196 ceph-mon[76349]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:38 smithi184 ceph-mon[100622]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:38 smithi184 ceph-mon[102844]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:38 smithi196 ceph-mon[76349]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:40 smithi184 ceph-mon[100622]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:40 smithi184 ceph-mon[102844]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:40 smithi196 ceph-mon[76349]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:41.259Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:41.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:41.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:12:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:43 smithi196 ceph-mon[76349]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[102844]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:43.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:43 smithi184 ceph-mon[100622]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:12:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:45 smithi184 ceph-mon[100622]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:45 smithi184 ceph-mon[102844]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:45 smithi196 ceph-mon[76349]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:46 smithi184 ceph-mon[100622]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:12:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:46 smithi184 ceph-mon[102844]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:12:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:46 smithi196 ceph-mon[76349]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:12:48.611 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 4h - - 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 5s ago 4h 92.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 4h 443M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 5s ago 4h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:12:49.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 5s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 5s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 5s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 5s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 5s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 5s ago 4h 72.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:12:49.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:12:49.140 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 5s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:12:49.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[100622]: from='client.61716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:49.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:49.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[100622]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:49.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[102844]: from='client.61716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:49.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:49.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:49 smithi184 ceph-mon[102844]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:49.461 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:12:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:12:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:12:49.462 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:12:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:49 smithi196 ceph-mon[76349]: from='client.61716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:49 smithi196 ceph-mon[76349]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:12:49.739 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:12:49.740 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:12:49.740 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:12:49.740 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:12:49.740 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:12:50.057 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:12:50.057 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:50.057 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:50.057 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:50.058 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:50 smithi196 ceph-mon[76349]: from='client.61722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:50 smithi196 ceph-mon[76349]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/982151210' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:50 smithi196 ceph-mon[76349]: from='client.61740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/787494946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:50.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[100622]: from='client.61722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[100622]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/982151210' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[100622]: from='client.61740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/787494946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[102844]: from='client.61722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[102844]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/982151210' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[102844]: from='client.61740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:50.678 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/787494946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:51 smithi184 ceph-mon[100622]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:51 smithi184 ceph-mon[102844]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:51.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:51.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:12:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:12:51.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:12:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:51 smithi196 ceph-mon[76349]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:12:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:12:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:52 smithi184 ceph-mon[102844]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:52 smithi184 ceph-mon[100622]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:52 smithi196 ceph-mon[76349]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:54 smithi184 ceph-mon[100622]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:54 smithi184 ceph-mon[102844]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:54 smithi196 ceph-mon[76349]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:56 smithi184 ceph-mon[100622]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:56 smithi184 ceph-mon[102844]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:56 smithi196 ceph-mon[76349]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:58 smithi184 ceph-mon[100622]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:58 smithi184 ceph-mon[102844]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:58 smithi196 ceph-mon[76349]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:00.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:00 smithi196 ceph-mon[76349]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:01.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:00 smithi184 ceph-mon[100622]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:01.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:00 smithi184 ceph-mon[102844]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:01.261Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:01.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:01.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[102844]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:03 smithi184 ceph-mon[100622]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:03.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:03 smithi196 ceph-mon[76349]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[102844]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:04 smithi184 ceph-mon[100622]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:04 smithi196 ceph-mon[76349]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:06 smithi184 ceph-mon[100622]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:06 smithi184 ceph-mon[102844]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:06 smithi196 ceph-mon[76349]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:08 smithi184 ceph-mon[100622]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:08 smithi184 ceph-mon[102844]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:08 smithi196 ceph-mon[76349]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:10 smithi184 ceph-mon[100622]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:10 smithi184 ceph-mon[102844]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:10 smithi196 ceph-mon[76349]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:11.451 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:11.261Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:11.451 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:11.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:11.451 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:11.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:12.250 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:13:12.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:12 smithi184 ceph-mon[100622]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:12.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:12 smithi184 ceph-mon[102844]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:12 smithi196 ceph-mon[76349]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:14.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:15 smithi184 ceph-mon[100622]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:15 smithi184 ceph-mon[102844]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:15.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:15 smithi196 ceph-mon[76349]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:16 smithi184 ceph-mon[100622]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:16 smithi184 ceph-mon[102844]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:17.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:16 smithi196 ceph-mon[76349]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:18 smithi184 ceph-mon[102844]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:18 smithi184 ceph-mon[100622]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:18 smithi196 ceph-mon[76349]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:20.354 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:13:20.884 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:13:20.884 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 4h 446M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 4h 310M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:13:20.885 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:13:20.886 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[100622]: from='client.61752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[100622]: from='client.71720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[100622]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[102844]: from='client.61752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[102844]: from='client.71720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:20 smithi184 ceph-mon[102844]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:20 smithi196 ceph-mon[76349]: from='client.61752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:20 smithi196 ceph-mon[76349]: from='client.71720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:20 smithi196 ceph-mon[76349]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:13:21.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:13:21.197 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:13:21.462 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:13:21.463 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:13:21.464 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:21.262Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:21.464 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:21.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:21.464 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:21.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:21.773 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:13:21.773 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:21.774 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:21.774 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:21.774 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:13:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[100622]: from='client.61764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/993612476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[100622]: from='client.61776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1710711660' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[102844]: from='client.61764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/993612476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[102844]: from='client.61776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1710711660' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:21 smithi196 ceph-mon[76349]: from='client.61764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/993612476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:21 smithi196 ceph-mon[76349]: from='client.61776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1710711660' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[100622]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:23 smithi184 ceph-mon[102844]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:23 smithi196 ceph-mon[76349]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[102844]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:24 smithi184 ceph-mon[100622]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:24 smithi196 ceph-mon[76349]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:26 smithi184 ceph-mon[102844]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:26 smithi184 ceph-mon[100622]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:26 smithi196 ceph-mon[76349]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:28 smithi184 ceph-mon[100622]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:28 smithi184 ceph-mon[102844]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:28 smithi196 ceph-mon[76349]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:31.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:30 smithi184 ceph-mon[100622]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:31.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:30 smithi184 ceph-mon[102844]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:30 smithi196 ceph-mon[76349]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:31.263Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:31.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:31.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:32.267 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:13:32.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:32 smithi184 ceph-mon[100622]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:32.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:32 smithi184 ceph-mon[102844]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:32 smithi196 ceph-mon[76349]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:34 smithi196 ceph-mon[76349]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:34 smithi184 ceph-mon[100622]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:34 smithi184 ceph-mon[102844]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:36 smithi196 ceph-mon[76349]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:36 smithi184 ceph-mon[100622]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:36 smithi184 ceph-mon[102844]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:38 smithi184 ceph-mon[100622]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:38 smithi184 ceph-mon[102844]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:39.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:39.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:38 smithi196 ceph-mon[76349]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:40 smithi196 ceph-mon[76349]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:41.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:40 smithi184 ceph-mon[100622]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:41.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:40 smithi184 ceph-mon[102844]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:41.264Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:41.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:41.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[100622]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:43 smithi184 ceph-mon[102844]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:43 smithi196 ceph-mon[76349]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:13:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:45 smithi184 ceph-mon[100622]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:45 smithi184 ceph-mon[102844]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:45 smithi196 ceph-mon[76349]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:46 smithi196 ceph-mon[76349]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:46 smithi184 ceph-mon[102844]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:46 smithi184 ceph-mon[100622]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:48 smithi184 ceph-mon[102844]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:48 smithi184 ceph-mon[100622]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:48 smithi196 ceph-mon[76349]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:50 smithi196 ceph-mon[76349]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:51.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:50 smithi184 ceph-mon[100622]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:51.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:50 smithi184 ceph-mon[102844]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:51.540 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:51.264Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:51.540 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:51.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:13:51.540 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:13:51.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:13:52.082 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:13:52.161 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:13:52] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 4h - - 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 4h - - 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 10s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 10s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:13:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 10s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 10s ago 4h 443M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 8s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 10s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 10s ago 4h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 10s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 10s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 4h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 4h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 8s ago 4h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 8s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 8s ago 4h 88.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 8s ago 4h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:13:52.662 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:13:52.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:13:52.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:13:52.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 8s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:13:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:13:52.993 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[102844]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[102844]: from='client.71756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[102844]: from='client.61800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[102844]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[100622]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[100622]: from='client.71756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[100622]: from='client.61800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:52 smithi184 ceph-mon[100622]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:52 smithi196 ceph-mon[76349]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:52 smithi196 ceph-mon[76349]: from='client.71756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:52 smithi196 ceph-mon[76349]: from='client.61800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:52 smithi196 ceph-mon[76349]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:53.270 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:13:53.270 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:13:53.270 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:13:53.271 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:13:53.609 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:13:53.609 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:53.609 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:54.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1061099081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:54.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[100622]: from='client.71774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:54.096 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/108417975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:54.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1061099081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:54.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[102844]: from='client.71774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:54.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/108417975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1061099081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:53 smithi196 ceph-mon[76349]: from='client.71774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/108417975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:55.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:54 smithi196 ceph-mon[76349]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:54 smithi184 ceph-mon[100622]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:54 smithi184 ceph-mon[102844]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:56.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:56 smithi196 ceph-mon[76349]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:56 smithi184 ceph-mon[100622]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:56 smithi184 ceph-mon[102844]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:58 smithi184 ceph-mon[100622]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:58 smithi184 ceph-mon[102844]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:59.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:59.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:58 smithi196 ceph-mon[76349]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:00 smithi196 ceph-mon[76349]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:01.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:00 smithi184 ceph-mon[100622]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:01.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:00 smithi184 ceph-mon[102844]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:01.265Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:01.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:01.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:14:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[100622]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:03 smithi184 ceph-mon[102844]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:03 smithi196 ceph-mon[76349]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[100622]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:05 smithi184 ceph-mon[102844]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:05 smithi196 ceph-mon[76349]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:06 smithi196 ceph-mon[76349]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:14:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:06 smithi184 ceph-mon[100622]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:14:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:06 smithi184 ceph-mon[102844]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:14:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:08 smithi196 ceph-mon[76349]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:08 smithi184 ceph-mon[100622]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:08 smithi184 ceph-mon[102844]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:10 smithi196 ceph-mon[76349]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:11.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:10 smithi184 ceph-mon[100622]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:11.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:10 smithi184 ceph-mon[102844]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:14:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:11.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:11.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:11.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:14:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:12 smithi196 ceph-mon[76349]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:14:13.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:12 smithi184 ceph-mon[100622]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:14:13.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:12 smithi184 ceph-mon[102844]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T00:14:14.053 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:14.053 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:15 smithi184 ceph-mon[100622]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:14:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:15 smithi184 ceph-mon[102844]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:14:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:15 smithi196 ceph-mon[76349]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:14:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:16 smithi196 ceph-mon[76349]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:14:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:16 smithi184 ceph-mon[100622]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:14:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:16 smithi184 ceph-mon[102844]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:14:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:18 smithi184 ceph-mon[102844]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:14:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:18 smithi184 ceph-mon[100622]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:14:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:18 smithi196 ceph-mon[76349]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:14:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:20 smithi196 ceph-mon[76349]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:14:21.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:20 smithi184 ceph-mon[100622]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:14:21.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:20 smithi184 ceph-mon[102844]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:14:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:21.267Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:21.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:21.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:22] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T00:14:23.907 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[102844]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[100622]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:23 smithi196 ceph-mon[76349]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T00:14:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:24.447 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 1s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 1s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 1s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 1s ago 4h 445M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 1s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 1s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 1s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:14:24.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 1s ago 4h 88.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 1s ago 4h 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 1s ago 4h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 0s ago 4h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 0s ago 4h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 0s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 1s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:14:24.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:14:24.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:14:24.786 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:14:25.074 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:14:25.074 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:14:25.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:14:25.386 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:14:25.386 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:25.387 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='client.61824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='client.71792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='client.71798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:14:25.387 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2490147490' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='client.61824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='client.71792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='client.71798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:14:25.388 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2490147490' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='client.61824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='client.71792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='client.71798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:14:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2490147490' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:26 smithi196 ceph-mon[76349]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3045184509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:26 smithi184 ceph-mon[102844]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3045184509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:26 smithi184 ceph-mon[100622]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3045184509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:27 smithi196 ceph-mon[76349]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:14:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:27 smithi184 ceph-mon[100622]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:14:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:27 smithi184 ceph-mon[102844]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:14:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:29 smithi196 ceph-mon[76349]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:14:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[100622]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[102844]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:14:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:30 smithi196 ceph-mon[76349]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:31.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:30 smithi184 ceph-mon[100622]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:31.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:30 smithi184 ceph-mon[102844]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:31.268Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:31.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:31.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:14:33.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:32 smithi184 ceph-mon[100622]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:33.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:32 smithi184 ceph-mon[102844]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:32 smithi196 ceph-mon[76349]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:14:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:34 smithi196 ceph-mon[76349]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T00:14:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:34 smithi184 ceph-mon[100622]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T00:14:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:34 smithi184 ceph-mon[102844]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T00:14:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:36 smithi196 ceph-mon[76349]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:14:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:36 smithi184 ceph-mon[100622]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:14:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:36 smithi184 ceph-mon[102844]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:14:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:38 smithi184 ceph-mon[100622]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:14:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:38 smithi184 ceph-mon[102844]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:14:39.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:39.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:38 smithi196 ceph-mon[76349]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:14:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:40 smithi196 ceph-mon[76349]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T00:14:41.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:40 smithi184 ceph-mon[100622]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T00:14:41.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:40 smithi184 ceph-mon[102844]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T00:14:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:41.268Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:41.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:41.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:14:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:42 smithi196 ceph-mon[76349]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:14:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:42 smithi184 ceph-mon[100622]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:14:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:42 smithi184 ceph-mon[102844]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:14:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:44 smithi196 ceph-mon[76349]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[100622]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:14:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:14:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:44 smithi184 ceph-mon[102844]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T00:14:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:46 smithi196 ceph-mon[76349]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:14:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:46 smithi184 ceph-mon[100622]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:14:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:46 smithi184 ceph-mon[102844]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:14:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:48 smithi184 ceph-mon[100622]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:48 smithi184 ceph-mon[102844]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:48 smithi196 ceph-mon[76349]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:50 smithi196 ceph-mon[76349]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:51.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:50 smithi184 ceph-mon[100622]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:51.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:50 smithi184 ceph-mon[102844]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:51.269Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:51.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:14:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:14:51.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:14:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:14:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:14:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:52 smithi196 ceph-mon[76349]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:52 smithi184 ceph-mon[100622]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:52 smithi184 ceph-mon[102844]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:54.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:54 smithi184 ceph-mon[100622]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:54.967 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:54 smithi184 ceph-mon[102844]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:54 smithi196 ceph-mon[76349]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:55.678 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:14:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:55 smithi184 ceph-mon[100622]: from='client.61860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:55 smithi184 ceph-mon[102844]: from='client.61860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:55 smithi196 ceph-mon[76349]: from='client.61860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 4h - - 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 13s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 11s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 13s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 11s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 13s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 13s ago 4h 446M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 11s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 13s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:14:56.215 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 13s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 11s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 13s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 13s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 13s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 13s ago 4h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 11s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 11s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 11s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 11s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 11s ago 4h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:14:56.216 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 13s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:14:56.217 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 11s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:14:56.527 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:56.528 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:14:56.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:14:56.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:14:56.530 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:14:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:14:56.798 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:14:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[100622]: from='client.71828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[100622]: from='client.61872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1998129774' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[100622]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[102844]: from='client.71828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[102844]: from='client.61872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1998129774' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:56 smithi184 ceph-mon[102844]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:57.107 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:14:57.107 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:57.108 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:57.108 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:57.108 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:56 smithi196 ceph-mon[76349]: from='client.71828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:56 smithi196 ceph-mon[76349]: from='client.61872 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1998129774' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:56 smithi196 ceph-mon[76349]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:57 smithi196 ceph-mon[76349]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1651690740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:57 smithi184 ceph-mon[100622]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1651690740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:57 smithi184 ceph-mon[102844]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1651690740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:58 smithi184 ceph-mon[100622]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:58 smithi184 ceph-mon[102844]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:58 smithi196 ceph-mon[76349]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:00 smithi196 ceph-mon[76349]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:01.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:00 smithi184 ceph-mon[100622]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:01.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:00 smithi184 ceph-mon[102844]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:01.269Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:01.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:01.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:02] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T00:15:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:02 smithi196 ceph-mon[76349]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:02 smithi184 ceph-mon[100622]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:02 smithi184 ceph-mon[102844]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:05 smithi196 ceph-mon[76349]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[100622]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:05 smithi184 ceph-mon[102844]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:06 smithi196 ceph-mon[76349]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:06 smithi184 ceph-mon[100622]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:06 smithi184 ceph-mon[102844]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:08 smithi184 ceph-mon[100622]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:08 smithi184 ceph-mon[102844]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:08 smithi196 ceph-mon[76349]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:10 smithi196 ceph-mon[76349]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:11.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:10 smithi184 ceph-mon[100622]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:11.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:10 smithi184 ceph-mon[102844]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:11.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:11.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:11.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:12] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T00:15:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:12 smithi196 ceph-mon[76349]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:12 smithi184 ceph-mon[100622]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:12 smithi184 ceph-mon[102844]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:15 smithi184 ceph-mon[100622]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:15 smithi184 ceph-mon[102844]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:15 smithi196 ceph-mon[76349]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:16 smithi184 ceph-mon[100622]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:16 smithi184 ceph-mon[102844]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:16 smithi196 ceph-mon[76349]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:18.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:18.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:18 smithi196 ceph-mon[76349]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:18 smithi184 ceph-mon[100622]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:18 smithi184 ceph-mon[102844]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:20 smithi196 ceph-mon[76349]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:21.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:20 smithi184 ceph-mon[100622]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:21.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:20 smithi184 ceph-mon[102844]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:21.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:21.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:21.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:15:23.135 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:22 smithi184 ceph-mon[100622]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:23.136 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:22 smithi184 ceph-mon[102844]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:22 smithi196 ceph-mon[76349]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[100622]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:25 smithi184 ceph-mon[102844]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:25 smithi196 ceph-mon[76349]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:26 smithi184 ceph-mon[100622]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:26 smithi184 ceph-mon[102844]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:26 smithi196 ceph-mon[76349]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:27.412 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:15:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:15:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-21T00:15:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 4h - - 2024-09-21T00:15:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 4s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 3s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 4s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 3s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 4s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 4s ago 4h 447M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 3s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 4s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 4s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 3s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 4s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 4s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 4s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 4s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:15:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 3s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 3s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 3s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 3s ago 4h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 4s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:15:27.951 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:15:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:27 smithi196 ceph-mon[76349]: from='client.71858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:27 smithi196 ceph-mon[76349]: from='client.71864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:15:28.263 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:15:28.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:27 smithi184 ceph-mon[100622]: from='client.71858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:27 smithi184 ceph-mon[100622]: from='client.71864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.263 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:27 smithi184 ceph-mon[102844]: from='client.71858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:27 smithi184 ceph-mon[102844]: from='client.71864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:28.532 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:15:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:15:28.532 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:15:28.533 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:15:28.848 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:15:28.848 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:28.848 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:28 smithi196 ceph-mon[76349]: from='client.71870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4063768288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:28 smithi196 ceph-mon[76349]: from='client.61920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:28 smithi196 ceph-mon[76349]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[100622]: from='client.71870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4063768288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[100622]: from='client.61920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[100622]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[102844]: from='client.71870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4063768288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[102844]: from='client.61920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[102844]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:30.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1643248201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:30.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1643248201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1643248201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:30 smithi196 ceph-mon[76349]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:31.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:30 smithi184 ceph-mon[100622]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:31.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:30 smithi184 ceph-mon[102844]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:31.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:31.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:31.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:15:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:32 smithi184 ceph-mon[100622]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:32 smithi184 ceph-mon[102844]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:32 smithi196 ceph-mon[76349]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:35.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:34 smithi184 ceph-mon[100622]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:35.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:34 smithi184 ceph-mon[102844]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:34 smithi196 ceph-mon[76349]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:36 smithi184 ceph-mon[100622]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:36 smithi184 ceph-mon[102844]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:36 smithi196 ceph-mon[76349]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:38 smithi184 ceph-mon[100622]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:38 smithi184 ceph-mon[102844]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:38 smithi196 ceph-mon[76349]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:40 smithi196 ceph-mon[76349]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:41.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:40 smithi184 ceph-mon[100622]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:41.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:40 smithi184 ceph-mon[102844]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:41.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:41.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:41.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:15:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:42 smithi196 ceph-mon[76349]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:43.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:42 smithi184 ceph-mon[100622]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:43.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:42 smithi184 ceph-mon[102844]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[100622]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[102844]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:45 smithi196 ceph-mon[76349]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:15:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:46 smithi196 ceph-mon[76349]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:46 smithi184 ceph-mon[100622]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:46 smithi184 ceph-mon[102844]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:48 smithi184 ceph-mon[100622]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:48 smithi184 ceph-mon[102844]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:48 smithi196 ceph-mon[76349]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:50 smithi196 ceph-mon[76349]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:51.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:50 smithi184 ceph-mon[100622]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:51.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:50 smithi184 ceph-mon[102844]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:51.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:51.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:15:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:15:51.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:15:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:15:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:15:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:52 smithi196 ceph-mon[76349]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:52 smithi184 ceph-mon[100622]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:52 smithi184 ceph-mon[102844]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:54 smithi196 ceph-mon[76349]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:54 smithi184 ceph-mon[100622]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:54 smithi184 ceph-mon[102844]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:56.827 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:56 smithi184 ceph-mon[100622]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:56.827 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:56 smithi184 ceph-mon[102844]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:56 smithi196 ceph-mon[76349]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:58.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:58 smithi196 ceph-mon[76349]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:58 smithi184 ceph-mon[100622]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:58 smithi184 ceph-mon[102844]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:59.137 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 16s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:15:59.669 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 16s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 16s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 16s ago 4h 447M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 16s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 16s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 16s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 16s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 16s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:15:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 14s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 14s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 73.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 16s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:15:59.671 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 14s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:15:59.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:15:59.981 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:15:59.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[100622]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[100622]: from='client.71900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[100622]: from='client.61944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[102844]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[102844]: from='client.71900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:59.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:15:59 smithi184 ceph-mon[102844]: from='client.61944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:59 smithi196 ceph-mon[76349]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:59 smithi196 ceph-mon[76349]: from='client.71900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:00.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:15:59 smithi196 ceph-mon[76349]: from='client.61944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:00.250 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:16:00.250 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:16:00.250 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:16:00.250 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:16:00.251 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:16:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:16:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1294072724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:00 smithi196 ceph-mon[76349]: from='client.61956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1040740698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:00 smithi196 ceph-mon[76349]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:01.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1294072724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:01.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[102844]: from='client.61956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:01.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1040740698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:01.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[102844]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:01.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1294072724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:01.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[100622]: from='client.61956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:01.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1040740698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:01.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:00 smithi184 ceph-mon[100622]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:01.274Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:01.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:01.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:16:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:02 smithi196 ceph-mon[76349]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:02 smithi184 ceph-mon[100622]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:02 smithi184 ceph-mon[102844]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:05 smithi196 ceph-mon[76349]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[100622]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[102844]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:05.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:06 smithi196 ceph-mon[76349]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:06 smithi184 ceph-mon[100622]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:06 smithi184 ceph-mon[102844]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:08 smithi184 ceph-mon[100622]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:08 smithi184 ceph-mon[102844]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:08 smithi196 ceph-mon[76349]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:10 smithi196 ceph-mon[76349]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:11.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:10 smithi184 ceph-mon[100622]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:11.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:10 smithi184 ceph-mon[102844]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:11.275Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:11.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:11.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:16:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:12 smithi196 ceph-mon[76349]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:12 smithi184 ceph-mon[100622]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:12 smithi184 ceph-mon[102844]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:15 smithi184 ceph-mon[100622]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:15 smithi184 ceph-mon[102844]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:15 smithi196 ceph-mon[76349]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:16.825 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:16 smithi184 ceph-mon[102844]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:16 smithi196 ceph-mon[76349]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:17.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:16 smithi184 ceph-mon[100622]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:19.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:18 smithi184 ceph-mon[100622]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:19.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:18 smithi184 ceph-mon[102844]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:18 smithi196 ceph-mon[76349]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:20 smithi196 ceph-mon[76349]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:21.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:20 smithi184 ceph-mon[100622]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:21.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:20 smithi184 ceph-mon[102844]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:21.276Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:21.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:21.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:16:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:22 smithi196 ceph-mon[76349]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:22 smithi184 ceph-mon[100622]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:22 smithi184 ceph-mon[102844]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:24.973 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[100622]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[102844]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:25 smithi196 ceph-mon[76349]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:25.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:26 smithi196 ceph-mon[76349]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:26 smithi184 ceph-mon[100622]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:26 smithi184 ceph-mon[102844]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:28 smithi184 ceph-mon[100622]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:28 smithi184 ceph-mon[102844]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:28 smithi196 ceph-mon[76349]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:30.863 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:16:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:30 smithi196 ceph-mon[76349]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:31.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:30 smithi184 ceph-mon[100622]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:31.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:30 smithi184 ceph-mon[102844]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:31.393 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:16:31.393 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 4h 449M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 4h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:16:31.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 4h 74.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:16:31.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:16:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:31.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:31.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:31.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:16:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:16:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:16:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:16:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:16:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:16:31.708 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:16:31.978 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:16:31.979 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:16:31.980 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[100622]: from='client.61968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[100622]: from='client.61974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[100622]: from='client.61980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2100252943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[102844]: from='client.61968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[102844]: from='client.61974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.980 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[102844]: from='client.61980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:31.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2100252943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:31 smithi196 ceph-mon[76349]: from='client.61968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:31 smithi196 ceph-mon[76349]: from='client.61974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:31 smithi196 ceph-mon[76349]: from='client.61980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2100252943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:32.290 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:16:32.290 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:32.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:32.291 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:32.291 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:32.291 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:16:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:32 smithi196 ceph-mon[76349]: from='client.71954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/687047210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:32 smithi196 ceph-mon[76349]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[100622]: from='client.71954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/687047210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[100622]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[102844]: from='client.71954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/687047210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:32 smithi184 ceph-mon[102844]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:35 smithi196 ceph-mon[76349]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:35 smithi184 ceph-mon[100622]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:35 smithi184 ceph-mon[102844]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:37 smithi196 ceph-mon[76349]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:37 smithi184 ceph-mon[100622]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:37 smithi184 ceph-mon[102844]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:39 smithi184 ceph-mon[100622]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:39 smithi184 ceph-mon[102844]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:40.123 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:39 smithi196 ceph-mon[76349]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:40.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:41.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:41.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:41.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:42.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:41 smithi196 ceph-mon[76349]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:41 smithi184 ceph-mon[100622]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:41 smithi184 ceph-mon[102844]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:16:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:44 smithi196 ceph-mon[76349]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[100622]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[102844]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[100622]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[102844]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:46 smithi196 ceph-mon[76349]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:16:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:48 smithi184 ceph-mon[100622]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:48 smithi184 ceph-mon[102844]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:48 smithi196 ceph-mon[76349]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:50 smithi184 ceph-mon[100622]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:50 smithi184 ceph-mon[102844]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:50 smithi196 ceph-mon[76349]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:51.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:51.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:16:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:16:51.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:16:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:52 smithi184 ceph-mon[100622]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:52 smithi184 ceph-mon[102844]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:16:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:52 smithi196 ceph-mon[76349]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:54 smithi184 ceph-mon[100622]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:54 smithi184 ceph-mon[102844]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:54 smithi196 ceph-mon[76349]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:56 smithi184 ceph-mon[100622]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:56 smithi184 ceph-mon[102844]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:56 smithi196 ceph-mon[76349]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:58 smithi184 ceph-mon[100622]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:58 smithi184 ceph-mon[102844]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:58 smithi196 ceph-mon[76349]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:00.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:00 smithi184 ceph-mon[100622]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:00.446 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:00.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:00 smithi184 ceph-mon[102844]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:00.446 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:00.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:00 smithi196 ceph-mon[76349]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:00.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:01.279Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:01.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:01.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:02 smithi184 ceph-mon[100622]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:02 smithi184 ceph-mon[102844]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:17:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:02 smithi196 ceph-mon[76349]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:02.598 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:17:03.148 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:17:03.148 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-21T00:17:03.148 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-21T00:17:03.148 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 4h 448M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:17:03.149 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 4h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:17:03.150 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 74.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:17:03.151 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:17:03.152 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:17:03.152 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:17:03.152 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:17:03.469 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:03 smithi184 ceph-mon[102844]: from='client.62004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:03.469 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:17:03.469 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:17:03.469 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:03.470 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:17:03.471 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:17:03.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:17:03.471 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:17:03.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:03 smithi184 ceph-mon[100622]: from='client.62004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:03 smithi196 ceph-mon[76349]: from='client.62004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:03.756 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:17:03.756 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:17:03.757 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:17:04.067 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:17:04.067 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:04.067 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='client.62010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1921179288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='client.71990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1689834684' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='client.62010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1921179288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='client.71990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1689834684' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='client.62010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1921179288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='client.71990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1689834684' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:06 smithi196 ceph-mon[76349]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[102844]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[100622]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:08 smithi196 ceph-mon[76349]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:08 smithi184 ceph-mon[100622]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:08 smithi184 ceph-mon[102844]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:10 smithi196 ceph-mon[76349]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:10 smithi184 ceph-mon[100622]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:10 smithi184 ceph-mon[102844]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:11.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:11.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:11.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:17:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:12 smithi196 ceph-mon[76349]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:12 smithi184 ceph-mon[100622]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:12 smithi184 ceph-mon[102844]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:14 smithi196 ceph-mon[76349]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:14 smithi184 ceph-mon[100622]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:14 smithi184 ceph-mon[102844]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:16 smithi196 ceph-mon[76349]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:16 smithi184 ceph-mon[100622]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:16 smithi184 ceph-mon[102844]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:18.595 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:18 smithi184 ceph-mon[100622]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:17:18.596 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:18 smithi184 ceph-mon[102844]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:17:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:18 smithi196 ceph-mon[76349]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:17:20.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:20 smithi184 ceph-mon[100622]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:20.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:20.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:20 smithi184 ceph-mon[102844]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:20.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:20 smithi196 ceph-mon[76349]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:21.281Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:21.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:21.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:22] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T00:17:22.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:22 smithi196 ceph-mon[76349]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:22 smithi184 ceph-mon[100622]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:22 smithi184 ceph-mon[102844]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:24 smithi196 ceph-mon[76349]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[100622]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[102844]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[100622]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[102844]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:26 smithi196 ceph-mon[76349]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:28 smithi184 ceph-mon[100622]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:28 smithi184 ceph-mon[102844]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:28 smithi196 ceph-mon[76349]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:30.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:30 smithi196 ceph-mon[76349]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:30.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:30 smithi184 ceph-mon[100622]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:30 smithi184 ceph-mon[102844]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:31.282Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:31.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:31.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:17:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:32 smithi184 ceph-mon[100622]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:32 smithi184 ceph-mon[102844]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:32 smithi196 ceph-mon[76349]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:34.363 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:17:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:34 smithi184 ceph-mon[100622]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:34 smithi184 ceph-mon[102844]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:34 smithi196 ceph-mon[76349]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 9s ago 4h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 4h 449M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 9s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:17:34.890 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 9s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 9s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 9s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 9s ago 4h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:17:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 9s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:17:35.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:17:35.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:17:35.204 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:17:35.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:17:35.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:17:35.204 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:17:35.472 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:17:35.473 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:17:35.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:17:35.786 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:17:35.786 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:35.786 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:35.786 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:35.786 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:35.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[102844]: from='client.62040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[102844]: from='client.62046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3252948003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:35.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[100622]: from='client.62040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[100622]: from='client.62046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3252948003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:35 smithi196 ceph-mon[76349]: from='client.62040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:35 smithi196 ceph-mon[76349]: from='client.62046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3252948003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[100622]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[100622]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[100622]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3149720076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[102844]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[102844]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[102844]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3149720076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:36 smithi196 ceph-mon[76349]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:36 smithi196 ceph-mon[76349]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:36 smithi196 ceph-mon[76349]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3149720076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:38 smithi184 ceph-mon[100622]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:38 smithi184 ceph-mon[102844]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:38 smithi196 ceph-mon[76349]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:40.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:40 smithi184 ceph-mon[102844]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:40.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:40.715 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:40 smithi184 ceph-mon[100622]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:40.715 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:40 smithi196 ceph-mon[76349]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:41.283Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:41.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:41.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:17:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:42 smithi184 ceph-mon[102844]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:42 smithi184 ceph-mon[100622]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:42 smithi196 ceph-mon[76349]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[100622]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[102844]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:44 smithi196 ceph-mon[76349]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:46 smithi196 ceph-mon[76349]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[102844]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[100622]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:17:48.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:48 smithi184 ceph-mon[100622]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:48.866 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:48 smithi184 ceph-mon[102844]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:48 smithi196 ceph-mon[76349]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:50 smithi196 ceph-mon[76349]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:50 smithi184 ceph-mon[100622]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:50 smithi184 ceph-mon[102844]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:51.283Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:51.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:17:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:17:51.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:17:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:17:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:17:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:52 smithi196 ceph-mon[76349]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:52 smithi184 ceph-mon[100622]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:52 smithi184 ceph-mon[102844]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:54 smithi196 ceph-mon[76349]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:54 smithi184 ceph-mon[100622]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:54 smithi184 ceph-mon[102844]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:56 smithi196 ceph-mon[76349]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:56 smithi184 ceph-mon[100622]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:56 smithi184 ceph-mon[102844]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:58.876 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:58 smithi184 ceph-mon[100622]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:58.877 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:58 smithi184 ceph-mon[102844]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:58 smithi196 ceph-mon[76349]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:00.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:00.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:00 smithi196 ceph-mon[76349]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:00.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:00 smithi184 ceph-mon[100622]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:00.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:00.922 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:00 smithi184 ceph-mon[102844]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:00.922 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:01.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:01.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:01.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:18:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:02 smithi184 ceph-mon[100622]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:02 smithi184 ceph-mon[102844]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:02 smithi196 ceph-mon[76349]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:04.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:04 smithi196 ceph-mon[76349]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:04.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:04.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:04.856 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[102844]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[100622]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:05.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.078 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:18:06.604 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 4h - - 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 4h 450M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 4h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:18:06.605 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 18.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 0s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 0s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 0s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 75.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:18:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:18:06.914 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:18:06.914 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:18:06.914 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:18:06.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:06.914 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:18:06.916 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:18:06.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[100622]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:06.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.917 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[102844]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:06.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.918 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:06.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:06 smithi196 ceph-mon[76349]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:07.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:18:07.182 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:18:07.183 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:18:07.184 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:18:07.184 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:18:07.184 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:18:07.184 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:18:07.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:18:07.491 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:18:07.492 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:07.492 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:07.492 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:07.492 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[100622]: from='client.62076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[100622]: from='client.62082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[100622]: from='client.72050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1725190294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3003518948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[102844]: from='client.62076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[102844]: from='client.62082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[102844]: from='client.72050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1725190294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3003518948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:07 smithi196 ceph-mon[76349]: from='client.62076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:07 smithi196 ceph-mon[76349]: from='client.62082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:07 smithi196 ceph-mon[76349]: from='client.72050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1725190294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3003518948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:08 smithi184 ceph-mon[100622]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:08 smithi184 ceph-mon[100622]: from='client.72062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:08 smithi184 ceph-mon[102844]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:08 smithi184 ceph-mon[102844]: from='client.72062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:08 smithi196 ceph-mon[76349]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:08 smithi196 ceph-mon[76349]: from='client.72062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:10 smithi184 ceph-mon[100622]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:10 smithi184 ceph-mon[102844]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:10 smithi196 ceph-mon[76349]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:11.285Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:11.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:11.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:18:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:12 smithi184 ceph-mon[100622]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:12 smithi184 ceph-mon[102844]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:12 smithi196 ceph-mon[76349]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:18:13.812803196Z level=info msg="Completed cleanup jobs" duration=13.128214ms 2024-09-21T00:18:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:14 smithi184 ceph-mon[100622]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:14 smithi184 ceph-mon[102844]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:15.092 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:14 smithi196 ceph-mon[76349]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:15.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:16 smithi184 ceph-mon[100622]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:16 smithi184 ceph-mon[102844]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:16 smithi196 ceph-mon[76349]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:18 smithi184 ceph-mon[100622]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:18 smithi184 ceph-mon[102844]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:18 smithi196 ceph-mon[76349]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:21.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:20 smithi184 ceph-mon[100622]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.040 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:21.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:20 smithi184 ceph-mon[102844]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.040 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:20 smithi196 ceph-mon[76349]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:21.286Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:21.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:21.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:21.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:22.150 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:18:23.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:22 smithi184 ceph-mon[100622]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:23.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:22 smithi184 ceph-mon[102844]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:22 smithi196 ceph-mon[76349]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:24.829 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:24 smithi196 ceph-mon[76349]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:24.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:24.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:24.830 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[100622]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[102844]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[100622]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[102844]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:26 smithi196 ceph-mon[76349]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:28 smithi184 ceph-mon[100622]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:28 smithi184 ceph-mon[102844]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:28 smithi196 ceph-mon[76349]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:30.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:30 smithi184 ceph-mon[100622]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:30 smithi184 ceph-mon[102844]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:30 smithi196 ceph-mon[76349]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:31.287Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:31.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:31.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:18:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:32 smithi184 ceph-mon[100622]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:32 smithi184 ceph-mon[102844]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:32 smithi196 ceph-mon[76349]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:34 smithi184 ceph-mon[100622]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:34 smithi184 ceph-mon[102844]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:34 smithi196 ceph-mon[76349]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:36 smithi184 ceph-mon[100622]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:36 smithi184 ceph-mon[102844]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:36 smithi196 ceph-mon[76349]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:37.785 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:18:38.310 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:18:38.310 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-21T00:18:38.310 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-21T00:18:38.310 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 4h 451M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 4h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:18:38.311 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 4h 75.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:18:38.312 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:18:38.621 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:18:38.622 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:18:38.623 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:18:38.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:18:38.623 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:18:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:18:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:18:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:18:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:18:38.894 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:18:38.894 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[100622]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:38.894 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[100622]: from='client.72074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:38.894 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/922167849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:38.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[102844]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:38.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[102844]: from='client.72074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:38.895 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/922167849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:38 smithi196 ceph-mon[76349]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:38 smithi196 ceph-mon[76349]: from='client.72074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/922167849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:39.207 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:18:39.207 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:39.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:39.207 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:39.208 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[100622]: from='client.72080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[100622]: from='client.72086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/803900512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[102844]: from='client.72080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[102844]: from='client.72086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/803900512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:39 smithi196 ceph-mon[76349]: from='client.72080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:39 smithi196 ceph-mon[76349]: from='client.72086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/803900512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[100622]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[100622]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[102844]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[102844]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:40 smithi196 ceph-mon[76349]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:40 smithi196 ceph-mon[76349]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:41.288Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:41.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:41.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:42.132 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:18:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:42 smithi184 ceph-mon[100622]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:42 smithi184 ceph-mon[102844]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:42 smithi196 ceph-mon[76349]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[100622]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[102844]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:44 smithi196 ceph-mon[76349]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[100622]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[102844]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:46 smithi196 ceph-mon[76349]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:18:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:48 smithi184 ceph-mon[100622]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:48 smithi184 ceph-mon[102844]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:48 smithi196 ceph-mon[76349]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:50 smithi184 ceph-mon[100622]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:50 smithi184 ceph-mon[102844]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:50 smithi196 ceph-mon[76349]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:51.289Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:51.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:18:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:18:51.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:18:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:51 smithi196 ceph-mon[76349]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:51 smithi184 ceph-mon[102844]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:51 smithi184 ceph-mon[100622]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:18:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:18:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:54 smithi184 ceph-mon[100622]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:54 smithi184 ceph-mon[102844]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:54 smithi196 ceph-mon[76349]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:56.701 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:56 smithi196 ceph-mon[76349]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:56 smithi184 ceph-mon[100622]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:56 smithi184 ceph-mon[102844]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:58 smithi184 ceph-mon[100622]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:58 smithi184 ceph-mon[102844]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:58 smithi196 ceph-mon[76349]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:00.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:00 smithi196 ceph-mon[76349]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:00.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:00 smithi184 ceph-mon[100622]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:00 smithi184 ceph-mon[102844]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:01.290Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:01.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:01.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:02] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:19:02.559 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:02 smithi184 ceph-mon[100622]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:02.560 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:02 smithi184 ceph-mon[102844]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:02 smithi196 ceph-mon[76349]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:04.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:04 smithi184 ceph-mon[100622]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:04.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:04 smithi184 ceph-mon[102844]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:04.775 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:04 smithi196 ceph-mon[76349]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:06 smithi196 ceph-mon[76349]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[100622]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[102844]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:08 smithi184 ceph-mon[100622]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:08 smithi184 ceph-mon[102844]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:08 smithi196 ceph-mon[76349]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:09.503 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 4h - - 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 4h - - 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 5s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 3s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 5s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:19:10.028 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 3s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 5s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 5s ago 4h 450M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 3s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 5s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 5s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 3s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 5s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 5s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 5s ago 4h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 5s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 3s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 3s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:19:10.029 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 3s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:19:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 3s ago 4h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:19:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:19:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:19:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:19:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:19:10.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:19:10.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:19:10.341 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:19:10.611 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:19:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:19:10.612 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:19:10.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[100622]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[100622]: from='client.62148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[100622]: from='client.62154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2321976417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[102844]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[102844]: from='client.62148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[102844]: from='client.62154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.613 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2321976417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:10 smithi196 ceph-mon[76349]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:10 smithi196 ceph-mon[76349]: from='client.62148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:10 smithi196 ceph-mon[76349]: from='client.62154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2321976417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:10.920 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:19:10.921 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:10.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:10.921 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:10.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[100622]: from='client.62160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[100622]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1628611701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[102844]: from='client.62160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[102844]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1628611701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:11.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:11.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:11.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:11 smithi196 ceph-mon[76349]: from='client.62160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:11 smithi196 ceph-mon[76349]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1628611701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:12] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:19:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:12 smithi184 ceph-mon[102844]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:12 smithi184 ceph-mon[100622]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:12 smithi196 ceph-mon[76349]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:14 smithi184 ceph-mon[100622]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:14 smithi184 ceph-mon[102844]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:14 smithi196 ceph-mon[76349]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:16 smithi184 ceph-mon[100622]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:16 smithi184 ceph-mon[102844]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:16 smithi196 ceph-mon[76349]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:18 smithi184 ceph-mon[100622]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:18 smithi184 ceph-mon[102844]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:18 smithi196 ceph-mon[76349]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:20 smithi196 ceph-mon[76349]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:20 smithi184 ceph-mon[100622]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:20 smithi184 ceph-mon[102844]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:21.292Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:21.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:21.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:22] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T00:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:22 smithi196 ceph-mon[76349]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:22 smithi184 ceph-mon[100622]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:22 smithi184 ceph-mon[102844]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:24 smithi196 ceph-mon[76349]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:24 smithi184 ceph-mon[100622]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:24 smithi184 ceph-mon[102844]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:25.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:26 smithi196 ceph-mon[76349]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[102844]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[100622]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:28 smithi196 ceph-mon[76349]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:28.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:28 smithi184 ceph-mon[102844]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:28.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:28 smithi184 ceph-mon[100622]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:31.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:30 smithi196 ceph-mon[76349]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:30 smithi184 ceph-mon[100622]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:30 smithi184 ceph-mon[102844]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:31.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:31.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:31.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:19:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:32 smithi184 ceph-mon[100622]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:32 smithi184 ceph-mon[102844]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:32 smithi196 ceph-mon[76349]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:34 smithi184 ceph-mon[100622]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:34 smithi184 ceph-mon[102844]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:34 smithi196 ceph-mon[76349]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:36 smithi184 ceph-mon[100622]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:36 smithi184 ceph-mon[102844]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:36 smithi196 ceph-mon[76349]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:38.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:38 smithi184 ceph-mon[100622]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:38.959 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:38 smithi184 ceph-mon[102844]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:38 smithi196 ceph-mon[76349]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:40 smithi184 ceph-mon[100622]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:40 smithi184 ceph-mon[102844]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:40 smithi196 ceph-mon[76349]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:41.219 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:19:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:41.294Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:41.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:41.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:41.745 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 4h - - 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 4h - - 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 17s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 17s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 15s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 17s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 17s ago 4h 450M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 15s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 17s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 17s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:19:41.746 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 15s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 17s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 17s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 17s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 17s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 15s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 15s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 15s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 15s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 15s ago 4h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 17s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:19:41.747 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 15s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:19:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:19:42.057 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:19:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:19:42.325 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:19:42.326 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:19:42.635 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:19:42.635 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:42.635 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:42.635 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:42.635 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: from='client.72152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: from='client.62196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3310316122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2115725034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: from='client.72152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: from='client.62196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3310316122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:43.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2115725034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: from='client.72152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: from='client.62196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3310316122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2115725034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:43.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:43 smithi184 ceph-mon[102844]: from='client.72170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:43.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:43 smithi184 ceph-mon[100622]: from='client.72170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:43 smithi196 ceph-mon[76349]: from='client.72170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:44.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:44 smithi184 ceph-mon[100622]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:44.934 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:44.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:44 smithi184 ceph-mon[102844]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:44.935 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:44 smithi196 ceph-mon[76349]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[100622]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[102844]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:46 smithi196 ceph-mon[76349]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:19:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:48 smithi196 ceph-mon[76349]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:48 smithi184 ceph-mon[100622]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:48 smithi184 ceph-mon[102844]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:50 smithi184 ceph-mon[100622]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:50 smithi184 ceph-mon[102844]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:50 smithi196 ceph-mon[76349]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:51.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:51.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:19:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:19:51.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:19:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:19:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:19:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:52 smithi184 ceph-mon[100622]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:52 smithi184 ceph-mon[102844]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:52 smithi196 ceph-mon[76349]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:54 smithi196 ceph-mon[76349]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:54 smithi184 ceph-mon[100622]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:54 smithi184 ceph-mon[102844]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:56 smithi196 ceph-mon[76349]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:56 smithi184 ceph-mon[100622]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:56 smithi184 ceph-mon[102844]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:58 smithi196 ceph-mon[76349]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:58 smithi184 ceph-mon[100622]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:58 smithi184 ceph-mon[102844]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:59 smithi196 ceph-mon[76349]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[102844]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[100622]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:01.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:01.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:01.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:01 smithi196 ceph-mon[76349]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:01 smithi184 ceph-mon[100622]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T00:20:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:01 smithi184 ceph-mon[102844]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:03 smithi184 ceph-mon[100622]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:03 smithi184 ceph-mon[102844]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:03 smithi196 ceph-mon[76349]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:06.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[100622]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[102844]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:05 smithi196 ceph-mon[76349]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:07.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:08.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:08 smithi196 ceph-mon[76349]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:08 smithi184 ceph-mon[100622]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:08 smithi184 ceph-mon[102844]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:10 smithi184 ceph-mon[100622]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:10 smithi184 ceph-mon[102844]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:10 smithi196 ceph-mon[76349]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:11.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:11.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:11.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T00:20:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:12 smithi184 ceph-mon[100622]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:12 smithi184 ceph-mon[102844]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:12 smithi196 ceph-mon[76349]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:12.931 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 4h - - 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 8s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 8s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 8s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 8s ago 4h 452M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:20:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 8s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 8s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 8s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 8s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 8s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 4h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 8s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:20:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:20:13.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:20:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:20:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:20:13.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:20:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:20:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:20:14.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:20:14.349 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:14.349 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:14.349 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='client.62220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='client.72188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='client.72194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/635340826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2897992083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='client.62220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='client.72188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='client.72194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/635340826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2897992083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='client.62220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='client.72188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='client.72194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/635340826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2897992083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:15 smithi184 ceph-mon[100622]: from='client.72206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:15 smithi184 ceph-mon[102844]: from='client.72206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:15 smithi196 ceph-mon[76349]: from='client.72206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:16 smithi184 ceph-mon[100622]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:16 smithi184 ceph-mon[102844]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:16 smithi196 ceph-mon[76349]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:18 smithi184 ceph-mon[100622]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:18 smithi184 ceph-mon[102844]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:18 smithi196 ceph-mon[76349]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:20 smithi196 ceph-mon[76349]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:20 smithi184 ceph-mon[100622]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:20 smithi184 ceph-mon[102844]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:21.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:21.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:21.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T00:20:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:22 smithi196 ceph-mon[76349]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:22 smithi184 ceph-mon[102844]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:22 smithi184 ceph-mon[100622]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:24 smithi196 ceph-mon[76349]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:25.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:24 smithi184 ceph-mon[100622]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:25.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:24 smithi184 ceph-mon[102844]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[100622]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[102844]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:26 smithi196 ceph-mon[76349]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:28 smithi196 ceph-mon[76349]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:29.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:28 smithi184 ceph-mon[100622]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:29.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:28 smithi184 ceph-mon[102844]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:30 smithi184 ceph-mon[100622]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:30 smithi184 ceph-mon[102844]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:30 smithi196 ceph-mon[76349]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:31.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:31.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:31.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:32] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:20:32.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:32 smithi196 ceph-mon[76349]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:32 smithi184 ceph-mon[100622]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:32 smithi184 ceph-mon[102844]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:34 smithi184 ceph-mon[100622]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:34 smithi184 ceph-mon[102844]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:34 smithi196 ceph-mon[76349]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:36 smithi184 ceph-mon[100622]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:36 smithi184 ceph-mon[102844]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:36 smithi196 ceph-mon[76349]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:39.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:38 smithi184 ceph-mon[100622]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:38 smithi184 ceph-mon[102844]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:38 smithi196 ceph-mon[76349]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:40 smithi184 ceph-mon[100622]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:40 smithi184 ceph-mon[102844]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:40 smithi196 ceph-mon[76349]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:41.298Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:41.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:41.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:42] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:20:42.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:42 smithi184 ceph-mon[100622]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:42.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:42 smithi184 ceph-mon[102844]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:42 smithi196 ceph-mon[76349]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:44.646 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:44 smithi184 ceph-mon[100622]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:44 smithi184 ceph-mon[102844]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:44 smithi196 ceph-mon[76349]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:45.184 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:20:45.184 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 4h - - 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 18s ago 4h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 18s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 4h 453M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 18s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:20:45.185 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 18s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 18s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 18s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 18s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 18s ago 4h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:20:45.186 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 18s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:20:45.507 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:20:45.508 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:20:45.793 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:20:45.794 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[100622]: from='client.62256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/800683878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[102844]: from='client.62256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:45.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/800683878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:46.106 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:46.106 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:46.106 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:46.106 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:46.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:45 smithi196 ceph-mon[76349]: from='client.62256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/800683878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:46.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[100622]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:46.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[100622]: from='client.62262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[100622]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[100622]: from='client.72242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/152951312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:46.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[102844]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:46.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[102844]: from='client.62262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[102844]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[102844]: from='client.72242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:46.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/152951312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:46 smithi196 ceph-mon[76349]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:46 smithi196 ceph-mon[76349]: from='client.62262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:46 smithi196 ceph-mon[76349]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:46 smithi196 ceph-mon[76349]: from='client.72242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/152951312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:48.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:20:49.027 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:48 smithi196 ceph-mon[76349]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:48 smithi184 ceph-mon[100622]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:48 smithi184 ceph-mon[102844]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:50 smithi196 ceph-mon[76349]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:50 smithi184 ceph-mon[102844]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:50 smithi184 ceph-mon[100622]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:51.298Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:51.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:20:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:20:51.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:20:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:20:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:20:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:52 smithi184 ceph-mon[100622]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:52 smithi184 ceph-mon[102844]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:52 smithi196 ceph-mon[76349]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:54 smithi184 ceph-mon[100622]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:54 smithi184 ceph-mon[102844]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:54 smithi196 ceph-mon[76349]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:56 smithi196 ceph-mon[76349]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:56 smithi184 ceph-mon[100622]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:56 smithi184 ceph-mon[102844]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:59.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:58 smithi184 ceph-mon[100622]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:59.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:58 smithi184 ceph-mon[102844]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:58 smithi196 ceph-mon[76349]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:00 smithi196 ceph-mon[76349]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:00 smithi184 ceph-mon[100622]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:00 smithi184 ceph-mon[102844]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:01.299Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:01.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:01.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:01 smithi196 ceph-mon[76349]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:01 smithi184 ceph-mon[100622]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:01 smithi184 ceph-mon[102844]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:02] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:21:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:03 smithi184 ceph-mon[100622]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:03 smithi184 ceph-mon[102844]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:03 smithi196 ceph-mon[76349]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[100622]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[102844]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:06 smithi196 ceph-mon[76349]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[100622]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[102844]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:07 smithi196 ceph-mon[76349]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:09 smithi196 ceph-mon[76349]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:09 smithi184 ceph-mon[100622]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:09 smithi184 ceph-mon[102844]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:11.299Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:11.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:11.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:11 smithi184 ceph-mon[100622]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:12] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:21:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:11 smithi184 ceph-mon[102844]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:11 smithi196 ceph-mon[76349]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:13 smithi184 ceph-mon[100622]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:13 smithi184 ceph-mon[102844]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:13 smithi196 ceph-mon[76349]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:16.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:15 smithi184 ceph-mon[100622]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:15 smithi184 ceph-mon[102844]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:15 smithi196 ceph-mon[76349]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:16.405 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 4h 454M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 4h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:21:16.934 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 4h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 4h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:21:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:21:16.936 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:17.242 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:21:17.243 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:21:17.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:16 smithi184 ceph-mon[102844]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:16 smithi184 ceph-mon[102844]: from='client.72260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.244 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:16 smithi184 ceph-mon[100622]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.244 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:16 smithi184 ceph-mon[100622]: from='client.72260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:16 smithi196 ceph-mon[76349]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:16 smithi196 ceph-mon[76349]: from='client.72260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:21:17.510 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:21:17.511 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:21:17.511 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:21:17.511 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:21:17.511 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:21:17.818 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:21:17.818 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:17.818 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[100622]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[100622]: from='client.72266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1256223396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[100622]: from='client.62316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3498535182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[102844]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[102844]: from='client.72266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1256223396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[102844]: from='client.62316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3498535182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:17 smithi196 ceph-mon[76349]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:17 smithi196 ceph-mon[76349]: from='client.72266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1256223396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:17 smithi196 ceph-mon[76349]: from='client.62316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3498535182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:19 smithi184 ceph-mon[100622]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:19 smithi184 ceph-mon[102844]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:19 smithi196 ceph-mon[76349]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:21.300Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:21.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:21.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:21 smithi184 ceph-mon[100622]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:21 smithi184 ceph-mon[102844]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:21:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:21 smithi196 ceph-mon[76349]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:23 smithi184 ceph-mon[100622]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:23 smithi184 ceph-mon[102844]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:23 smithi196 ceph-mon[76349]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[102844]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[100622]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:26 smithi196 ceph-mon[76349]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[100622]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[102844]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:28 smithi196 ceph-mon[76349]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:29.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:29.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:29.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:30.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:30 smithi184 ceph-mon[100622]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:30.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:30.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:30 smithi184 ceph-mon[102844]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:30.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:30.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:30 smithi196 ceph-mon[76349]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:30.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:31.301Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:31.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:31.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:32 smithi184 ceph-mon[100622]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:32 smithi184 ceph-mon[102844]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:21:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:32 smithi196 ceph-mon[76349]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:34 smithi184 ceph-mon[100622]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:34 smithi184 ceph-mon[102844]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:34 smithi196 ceph-mon[76349]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:36 smithi196 ceph-mon[76349]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:36 smithi184 ceph-mon[100622]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:36 smithi184 ceph-mon[102844]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:38 smithi196 ceph-mon[76349]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:38 smithi184 ceph-mon[100622]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:38 smithi184 ceph-mon[102844]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:40.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:40 smithi184 ceph-mon[100622]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:40.306 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:40.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:40 smithi184 ceph-mon[102844]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:40.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:40 smithi196 ceph-mon[76349]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:41.301Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:41.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:41.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:42 smithi184 ceph-mon[100622]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:42 smithi184 ceph-mon[102844]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:21:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:42 smithi196 ceph-mon[76349]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:44 smithi196 ceph-mon[76349]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:44 smithi184 ceph-mon[100622]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:44 smithi184 ceph-mon[102844]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[100622]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[102844]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:46 smithi196 ceph-mon[76349]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.124 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[102844]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[100622]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:48 smithi196 ceph-mon[76349]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:21:48.669 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:21:48.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-21T00:21:48.669 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 4h - - 2024-09-21T00:21:48.669 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 1s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 1s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 4h 454M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 1s ago 4h 310M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:21:48.670 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 1s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 1s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 1s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 1s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 1s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:21:48.671 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 1s ago 4h 75.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:21:48.672 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:21:48.672 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:21:48.672 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:21:48.672 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 1s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:21:48.985 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:21:48.985 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:21:48.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:21:48.986 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:21:49.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:21:49.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:21:49.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[100622]: from='client.62328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[100622]: from='client.62334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[100622]: from='client.62340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2998166657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[102844]: from='client.62328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[102844]: from='client.62334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[102844]: from='client.62340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2998166657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:49.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:49 smithi196 ceph-mon[76349]: from='client.62328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:49 smithi196 ceph-mon[76349]: from='client.62334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:49 smithi196 ceph-mon[76349]: from='client.62340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:49.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2998166657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:49.588 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:21:49.589 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:49.589 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:49.589 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:49.589 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:50 smithi196 ceph-mon[76349]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:50 smithi196 ceph-mon[76349]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/449430649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:50.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[100622]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:50.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:50.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[100622]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:50.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/449430649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:50.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[102844]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:50.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:50.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[102844]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:50.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/449430649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:51.302Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:51.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:21:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:21:51.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:21:52.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:52 smithi196 ceph-mon[76349]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:52 smithi184 ceph-mon[100622]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:52 smithi184 ceph-mon[102844]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:21:52] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:21:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:54 smithi184 ceph-mon[100622]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:54 smithi184 ceph-mon[102844]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:54 smithi196 ceph-mon[76349]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:56 smithi184 ceph-mon[100622]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:56 smithi184 ceph-mon[102844]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:56 smithi196 ceph-mon[76349]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:58 smithi184 ceph-mon[100622]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:58 smithi184 ceph-mon[102844]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:58 smithi196 ceph-mon[76349]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:00 smithi184 ceph-mon[100622]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:00 smithi184 ceph-mon[102844]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:00 smithi196 ceph-mon[76349]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:01.303Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:01.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:01.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:02 smithi184 ceph-mon[100622]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:02 smithi184 ceph-mon[102844]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:22:02.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:02 smithi196 ceph-mon[76349]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:04 smithi184 ceph-mon[100622]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:04 smithi184 ceph-mon[102844]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:04 smithi196 ceph-mon[76349]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[100622]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[102844]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:06 smithi196 ceph-mon[76349]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[100622]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[102844]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:08 smithi196 ceph-mon[76349]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:10 smithi184 ceph-mon[100622]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:10 smithi184 ceph-mon[102844]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:10.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:10 smithi196 ceph-mon[76349]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:10.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:11.304Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:11.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:11.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:12.293 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:22:12.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:12 smithi196 ceph-mon[76349]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:12 smithi184 ceph-mon[100622]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:12 smithi184 ceph-mon[102844]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:14 smithi184 ceph-mon[102844]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:14 smithi184 ceph-mon[100622]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:14 smithi196 ceph-mon[76349]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:16 smithi196 ceph-mon[76349]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:16 smithi184 ceph-mon[100622]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:16 smithi184 ceph-mon[102844]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:18 smithi196 ceph-mon[76349]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:18 smithi184 ceph-mon[100622]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:18 smithi184 ceph-mon[102844]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:19.884 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 4h - - 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 13s ago 4h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 13s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 4h 455M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:22:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 13s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 4h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 13s ago 4h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 13s ago 4h 75.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:22:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:22:20.412 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:22:20.412 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:22:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:20 smithi196 ceph-mon[76349]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:22:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:22:20.733 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:22:20.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:20 smithi184 ceph-mon[100622]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:20.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:20.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:20 smithi184 ceph-mon[102844]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:20.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:21.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:22:21.002 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:22:21.002 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:22:21.002 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:22:21.003 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:22:21.316 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:22:21.316 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:21.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:21.316 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:21.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:21 smithi196 ceph-mon[76349]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:21 smithi196 ceph-mon[76349]: from='client.62370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:21 smithi196 ceph-mon[76349]: from='client.62376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/516971964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[100622]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[100622]: from='client.62370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[100622]: from='client.62376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/516971964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[102844]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[102844]: from='client.62370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[102844]: from='client.62376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:21.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/516971964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:21.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:21.304Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:21.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:21.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:21.800 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:21.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:22:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:22 smithi196 ceph-mon[76349]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:22 smithi196 ceph-mon[76349]: from='client.62388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4249884439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[100622]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[100622]: from='client.62388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4249884439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[102844]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[102844]: from='client.62388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4249884439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:24 smithi196 ceph-mon[76349]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:24 smithi184 ceph-mon[102844]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:24 smithi184 ceph-mon[100622]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[102844]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[100622]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:26 smithi196 ceph-mon[76349]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[102844]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[100622]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:28 smithi196 ceph-mon[76349]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:29.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:29.746 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:30 smithi184 ceph-mon[100622]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:30 smithi184 ceph-mon[102844]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:30 smithi196 ceph-mon[76349]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:31.611 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:31.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:31.611 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:31.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:31.612 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:31.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:22:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:32 smithi196 ceph-mon[76349]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:32.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:32 smithi184 ceph-mon[102844]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:32.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:32 smithi184 ceph-mon[100622]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:34 smithi184 ceph-mon[100622]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:34 smithi184 ceph-mon[102844]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:34 smithi196 ceph-mon[76349]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:36 smithi184 ceph-mon[100622]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:36 smithi184 ceph-mon[102844]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:36 smithi196 ceph-mon[76349]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:38 smithi184 ceph-mon[100622]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:38 smithi184 ceph-mon[102844]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:38 smithi196 ceph-mon[76349]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:40 smithi184 ceph-mon[100622]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:40 smithi184 ceph-mon[102844]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:40 smithi196 ceph-mon[76349]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:41.306Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:41.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:41.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:22:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:42 smithi184 ceph-mon[100622]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:42 smithi184 ceph-mon[102844]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:42 smithi196 ceph-mon[76349]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:44 smithi184 ceph-mon[100622]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:44 smithi184 ceph-mon[102844]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:44 smithi196 ceph-mon[76349]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[100622]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[102844]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:46 smithi196 ceph-mon[76349]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[100622]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[102844]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:48 smithi196 ceph-mon[76349]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:22:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:50 smithi184 ceph-mon[100622]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:50 smithi184 ceph-mon[102844]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:50 smithi196 ceph-mon[76349]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:51.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:51.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:22:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:22:51.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:22:51.616 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 93.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:22:52.179 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 4h 456M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 4h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:22:52.180 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 4h 75.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:22:52.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:22:52.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:22:52.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:22:52.181 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:22:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:22:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:22:52.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:22:52.504 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:22:52.797 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:22:52.798 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:22:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[100622]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[100622]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1937822117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[102844]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[102844]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1937822117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:52.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:52 smithi196 ceph-mon[76349]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:52.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:52 smithi196 ceph-mon[76349]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:52.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1937822117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:53.131 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:22:53.131 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:53.132 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:53.132 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:53 smithi196 ceph-mon[76349]: from='client.72368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:53 smithi196 ceph-mon[76349]: from='client.62412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:53 smithi196 ceph-mon[76349]: from='client.62424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2678722031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[102844]: from='client.72368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[102844]: from='client.62412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[102844]: from='client.62424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2678722031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[100622]: from='client.72368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[100622]: from='client.62412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[100622]: from='client.62424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2678722031' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:54 smithi196 ceph-mon[76349]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:54 smithi184 ceph-mon[102844]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:54 smithi184 ceph-mon[100622]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:56 smithi196 ceph-mon[76349]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:56 smithi184 ceph-mon[102844]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:56 smithi184 ceph-mon[100622]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:58 smithi196 ceph-mon[76349]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:58 smithi184 ceph-mon[102844]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:58 smithi184 ceph-mon[100622]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:00 smithi196 ceph-mon[76349]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:00 smithi184 ceph-mon[100622]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:00 smithi184 ceph-mon[102844]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:01.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:01.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:01.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:02] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:23:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:02 smithi196 ceph-mon[76349]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:02 smithi184 ceph-mon[100622]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:02 smithi184 ceph-mon[102844]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:04 smithi196 ceph-mon[76349]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:04 smithi184 ceph-mon[100622]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:04 smithi184 ceph-mon[102844]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:06 smithi196 ceph-mon[76349]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[100622]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[102844]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[100622]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[102844]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:08 smithi196 ceph-mon[76349]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:10.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:10 smithi184 ceph-mon[100622]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:10.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:10.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:10 smithi184 ceph-mon[102844]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:10.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:10 smithi196 ceph-mon[76349]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:11.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:11.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:11.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:11.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:11.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:11.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:12] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:23:12.774 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:12 smithi196 ceph-mon[76349]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:13.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:12 smithi184 ceph-mon[100622]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:13.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:12 smithi184 ceph-mon[102844]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:14 smithi196 ceph-mon[76349]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:14 smithi184 ceph-mon[100622]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:14 smithi184 ceph-mon[102844]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:16 smithi184 ceph-mon[100622]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:16 smithi184 ceph-mon[102844]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:16 smithi196 ceph-mon[76349]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:18 smithi184 ceph-mon[100622]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:18 smithi184 ceph-mon[102844]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:18 smithi196 ceph-mon[76349]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:20 smithi184 ceph-mon[100622]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:20 smithi184 ceph-mon[102844]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:20 smithi196 ceph-mon[76349]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:21.308Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:21.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:21.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:23:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:22 smithi184 ceph-mon[100622]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:22 smithi184 ceph-mon[102844]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:22 smithi196 ceph-mon[76349]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:23.425 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 4h - - 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 17s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 17s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 17s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 17s ago 4h 456M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:23:23.954 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 17s ago 4h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 17s ago 4h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 17s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 17s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 17s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 17s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 4h 88.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 4h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 17s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:23:23.955 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:23:24.264 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:24.265 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:23:24.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:23:24.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:23:24.537 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[102844]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[102844]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[102844]: from='client.62442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3852108991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[100622]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[100622]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[100622]: from='client.62442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3852108991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:24.849 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:23:24.850 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:24.850 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:24.850 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:24.850 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:24 smithi196 ceph-mon[76349]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:23:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:24 smithi196 ceph-mon[76349]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:24 smithi196 ceph-mon[76349]: from='client.62442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3852108991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[100622]: from='client.62448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[100622]: from='client.62460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/904478229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[102844]: from='client.62448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[102844]: from='client.62460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/904478229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:25 smithi196 ceph-mon[76349]: from='client.62448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:25 smithi196 ceph-mon[76349]: from='client.62460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/904478229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[100622]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[102844]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:26 smithi196 ceph-mon[76349]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[100622]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[102844]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:28 smithi196 ceph-mon[76349]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:30 smithi196 ceph-mon[76349]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:31.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:30 smithi184 ceph-mon[100622]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:31.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:30 smithi184 ceph-mon[102844]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:31.506 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:31.308Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:31.506 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:31.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:31.506 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:31.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:23:32.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:32 smithi196 ceph-mon[76349]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:32.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:32 smithi184 ceph-mon[100622]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:32.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:32 smithi184 ceph-mon[102844]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:35.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:34 smithi196 ceph-mon[76349]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:34 smithi184 ceph-mon[100622]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:34 smithi184 ceph-mon[102844]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:36 smithi196 ceph-mon[76349]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:36 smithi184 ceph-mon[100622]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:36 smithi184 ceph-mon[102844]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:38 smithi196 ceph-mon[76349]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:39.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:38 smithi184 ceph-mon[100622]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:39.167 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:38 smithi184 ceph-mon[102844]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:40 smithi196 ceph-mon[76349]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:40 smithi184 ceph-mon[102844]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:40 smithi184 ceph-mon[100622]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:41.309Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:41.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:41.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:23:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:42 smithi196 ceph-mon[76349]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:42 smithi184 ceph-mon[100622]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:42 smithi184 ceph-mon[102844]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:44 smithi184 ceph-mon[100622]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:44 smithi184 ceph-mon[102844]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:44 smithi196 ceph-mon[76349]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:45.222 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:46 smithi196 ceph-mon[76349]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[100622]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[102844]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:48 smithi196 ceph-mon[76349]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[100622]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[102844]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:49.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:23:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:50 smithi196 ceph-mon[76349]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:50 smithi184 ceph-mon[100622]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:50 smithi184 ceph-mon[102844]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:51.310Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:51.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:23:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:23:51.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:23:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:23:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:23:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:52 smithi184 ceph-mon[102844]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:52 smithi184 ceph-mon[100622]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:52 smithi196 ceph-mon[76349]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:55.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:54 smithi196 ceph-mon[76349]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:55.134 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:54 smithi184 ceph-mon[100622]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:55.134 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:54 smithi184 ceph-mon[102844]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:55.135 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:23:55.660 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:23:55.660 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 4h - - 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 4h - - 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 9s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 7s ago 4h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 9s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 9s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 4h 457M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:23:55.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 9s ago 4h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 7s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 9s ago 4h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 9s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 9s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 9s ago 4h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 4h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 7s ago 4h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:23:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:23:55.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:23:55.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 9s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:23:55.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:23:55.972 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:23:55.972 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:23:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:23:55.974 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:23:56.243 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:23:56.244 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:23:56.244 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:23:56.244 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:23:56.244 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:23:56.556 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:23:56.556 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:56.556 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:56.556 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:56.556 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: from='client.72434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: from='client.62478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: from='client.62484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/448212856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3328114189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: from='client.72434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: from='client.62478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: from='client.62484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/448212856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3328114189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: from='client.72434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: from='client.62478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: from='client.62484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/448212856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3328114189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:57 smithi196 ceph-mon[76349]: from='client.72458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:57 smithi184 ceph-mon[100622]: from='client.72458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:57 smithi184 ceph-mon[102844]: from='client.72458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:58 smithi196 ceph-mon[76349]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.188 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:58 smithi184 ceph-mon[100622]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:58 smithi184 ceph-mon[102844]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:59 smithi196 ceph-mon[76349]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:23:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[102844]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[100622]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:23:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:01.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:01.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:01.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:02.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:01 smithi184 ceph-mon[102844]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:01 smithi184 ceph-mon[100622]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:24:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:01 smithi196 ceph-mon[76349]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:03 smithi196 ceph-mon[76349]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:03 smithi184 ceph-mon[100622]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:03 smithi184 ceph-mon[102844]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:05 smithi184 ceph-mon[100622]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:05 smithi184 ceph-mon[102844]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:05 smithi196 ceph-mon[76349]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[100622]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:24:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[102844]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:24:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:08 smithi196 ceph-mon[76349]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:24:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:10 smithi184 ceph-mon[100622]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:10 smithi184 ceph-mon[102844]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:10 smithi196 ceph-mon[76349]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:11.312Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:11.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:11.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:24:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:12 smithi184 ceph-mon[100622]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:12 smithi184 ceph-mon[102844]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:12 smithi196 ceph-mon[76349]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:24:14.758 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:14 smithi196 ceph-mon[76349]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:24:14.758 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:14 smithi184 ceph-mon[102844]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:24:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:14 smithi184 ceph-mon[100622]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:24:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:16.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:16 smithi196 ceph-mon[76349]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:24:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:16 smithi184 ceph-mon[100622]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:24:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:16 smithi184 ceph-mon[102844]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:24:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:18 smithi184 ceph-mon[100622]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:24:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:18 smithi184 ceph-mon[102844]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:24:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:18 smithi196 ceph-mon[76349]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:24:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:20 smithi184 ceph-mon[100622]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:24:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:20 smithi184 ceph-mon[102844]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:24:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:20 smithi196 ceph-mon[76349]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:24:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:21.312Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:21.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:21.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:24:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:22 smithi184 ceph-mon[100622]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:24:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:22 smithi184 ceph-mon[102844]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:24:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:22 smithi196 ceph-mon[76349]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:24:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:24 smithi184 ceph-mon[100622]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:24:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:24 smithi184 ceph-mon[102844]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:24:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:24 smithi196 ceph-mon[76349]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:24:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:26 smithi184 ceph-mon[102844]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:24:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:26 smithi184 ceph-mon[100622]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:24:26.844 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:24:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:26 smithi196 ceph-mon[76349]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 4h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 4h 458M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 4h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:24:27.369 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 4h 86.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 4h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 4h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 4h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 4h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:24:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:24:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:24:27.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:24:27.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[100622]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:27 smithi184 ceph-mon[102844]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:27 smithi196 ceph-mon[76349]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:27.953 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:24:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:24:27.955 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:24:28.275 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:24:28.275 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:28.275 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:28.275 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:28.275 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='client.62514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='client.62520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1478561550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3846927193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='client.62514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='client.62520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1478561550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3846927193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='client.62514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='client.62520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1478561550' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:29.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3846927193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:29 smithi196 ceph-mon[76349]: from='client.72494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:29 smithi184 ceph-mon[100622]: from='client.72494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:29 smithi184 ceph-mon[102844]: from='client.72494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:30 smithi196 ceph-mon[76349]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T00:24:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:30 smithi184 ceph-mon[100622]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T00:24:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:30 smithi184 ceph-mon[102844]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T00:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:31.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:31.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:31.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:24:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:32 smithi196 ceph-mon[76349]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:24:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:32 smithi184 ceph-mon[100622]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:24:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:32 smithi184 ceph-mon[102844]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:24:34.868 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:34 smithi184 ceph-mon[100622]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:24:34.868 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:34 smithi184 ceph-mon[102844]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:24:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:34 smithi196 ceph-mon[76349]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:24:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:36 smithi196 ceph-mon[76349]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:24:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:36 smithi184 ceph-mon[100622]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:24:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:36 smithi184 ceph-mon[102844]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:24:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:38 smithi196 ceph-mon[76349]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:24:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:38 smithi184 ceph-mon[100622]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:24:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:38 smithi184 ceph-mon[102844]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:24:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:40 smithi196 ceph-mon[76349]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:24:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:40 smithi184 ceph-mon[102844]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:24:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:40 smithi184 ceph-mon[100622]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:24:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:41.314Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:41.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:41.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:24:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:42 smithi196 ceph-mon[76349]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:24:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:42 smithi184 ceph-mon[100622]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:24:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:42 smithi184 ceph-mon[102844]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:24:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:44 smithi196 ceph-mon[76349]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:24:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:44 smithi184 ceph-mon[100622]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:24:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:44 smithi184 ceph-mon[102844]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:24:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:46 smithi196 ceph-mon[76349]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:24:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:46 smithi184 ceph-mon[100622]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:24:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:46 smithi184 ceph-mon[102844]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[100622]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[102844]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:48 smithi196 ceph-mon[76349]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T00:24:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:24:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:50 smithi184 ceph-mon[100622]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:50 smithi184 ceph-mon[102844]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:50 smithi196 ceph-mon[76349]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:51.315Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:51.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:24:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:24:51.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:24:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:24:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:24:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:52 smithi184 ceph-mon[100622]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:52 smithi184 ceph-mon[102844]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:52 smithi196 ceph-mon[76349]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:55.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:54 smithi184 ceph-mon[100622]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:55.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:54 smithi184 ceph-mon[102844]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:54 smithi196 ceph-mon[76349]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:56 smithi184 ceph-mon[100622]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:56 smithi184 ceph-mon[102844]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:56 smithi196 ceph-mon[76349]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:58.559 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:24:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:58 smithi184 ceph-mon[100622]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:58 smithi184 ceph-mon[102844]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 12s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 12s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 12s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 12s ago 4h 459M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:24:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 12s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 12s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 12s ago 4h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 12s ago 4h 76.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 12s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 4h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 4h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 4h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 4h 77.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 12s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:24:59.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:24:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:58 smithi196 ceph-mon[76349]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:59.399 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:24:59.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:24:59.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:24:59.401 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:24:59.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:24:59.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:24:59.401 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:24:59.667 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:24:59.667 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:24:59.668 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:24:59.982 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:24:59.982 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:59.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:59.982 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:59.982 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[100622]: from='client.62544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[100622]: from='client.72512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3680278388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[102844]: from='client.62544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[102844]: from='client.72512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:59.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:24:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3680278388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:59 smithi196 ceph-mon[76349]: from='client.62544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:59 smithi196 ceph-mon[76349]: from='client.72512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:24:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3680278388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[100622]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[100622]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[100622]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4285908612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[102844]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[102844]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[102844]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4285908612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:00 smithi196 ceph-mon[76349]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:00 smithi196 ceph-mon[76349]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:00 smithi196 ceph-mon[76349]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4285908612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:01.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:01.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:01.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:25:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:02 smithi184 ceph-mon[100622]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:02 smithi184 ceph-mon[102844]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:02 smithi196 ceph-mon[76349]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:04 smithi184 ceph-mon[100622]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:04 smithi184 ceph-mon[102844]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:05.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:04 smithi196 ceph-mon[76349]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:06 smithi184 ceph-mon[100622]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:06 smithi184 ceph-mon[102844]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:06 smithi196 ceph-mon[76349]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:08 smithi196 ceph-mon[76349]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[100622]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[102844]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:09.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:10 smithi196 ceph-mon[76349]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:10 smithi184 ceph-mon[100622]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:10 smithi184 ceph-mon[102844]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:11.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:11.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:11.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:25:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:12 smithi196 ceph-mon[76349]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:12 smithi184 ceph-mon[100622]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:12 smithi184 ceph-mon[102844]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:13 smithi184 ceph-mon[100622]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:13 smithi184 ceph-mon[102844]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:13 smithi196 ceph-mon[76349]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:15.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:15.969 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:15 smithi184 ceph-mon[100622]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:15.969 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:15 smithi184 ceph-mon[102844]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:16.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:15 smithi196 ceph-mon[76349]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:18 smithi184 ceph-mon[100622]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:18 smithi184 ceph-mon[102844]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:18.335 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:18 smithi196 ceph-mon[76349]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:20 smithi184 ceph-mon[100622]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:20 smithi184 ceph-mon[102844]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:20 smithi196 ceph-mon[76349]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:21.317Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:21.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:21.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:22 smithi184 ceph-mon[100622]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:22 smithi184 ceph-mon[102844]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:25:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:22 smithi196 ceph-mon[76349]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:24 smithi184 ceph-mon[100622]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:24 smithi184 ceph-mon[102844]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:24 smithi196 ceph-mon[76349]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:26 smithi184 ceph-mon[100622]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:26 smithi184 ceph-mon[102844]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:26 smithi196 ceph-mon[76349]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[100622]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[102844]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:27 smithi196 ceph-mon[76349]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:30.270 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 4h 460M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:25:30.796 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 4h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 77.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:25:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:25:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:30 smithi196 ceph-mon[76349]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:30 smithi184 ceph-mon[100622]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:30 smithi184 ceph-mon[102844]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:31.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:25:31.106 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:25:31.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:25:31.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:25:31.107 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:25:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:25:31.379 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:25:31.379 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:31.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:31.380 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:31.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:31.380 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:31.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:31.689 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:25:31.689 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:31.689 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:31.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[102844]: from='client.72542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[102844]: from='client.62586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[102844]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2784591302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:31.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[100622]: from='client.72542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[100622]: from='client.62586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[100622]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.690 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2784591302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:31 smithi196 ceph-mon[76349]: from='client.72542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:31 smithi196 ceph-mon[76349]: from='client.62586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:31 smithi196 ceph-mon[76349]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2784591302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:25:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:32 smithi196 ceph-mon[76349]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:32 smithi196 ceph-mon[76349]: from='client.72566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2724447115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[100622]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[100622]: from='client.72566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2724447115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[102844]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[102844]: from='client.72566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2724447115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:34 smithi196 ceph-mon[76349]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:34 smithi184 ceph-mon[102844]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:34 smithi184 ceph-mon[100622]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:36.720 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:36 smithi196 ceph-mon[76349]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:36 smithi184 ceph-mon[100622]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:36 smithi184 ceph-mon[102844]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:38 smithi196 ceph-mon[76349]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:38 smithi184 ceph-mon[100622]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:38 smithi184 ceph-mon[102844]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:40 smithi196 ceph-mon[76349]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:40 smithi184 ceph-mon[100622]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:40 smithi184 ceph-mon[102844]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:41.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:41.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:41.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:25:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:42 smithi196 ceph-mon[76349]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:42 smithi184 ceph-mon[100622]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:42 smithi184 ceph-mon[102844]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:44 smithi196 ceph-mon[76349]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:44 smithi184 ceph-mon[100622]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:44 smithi184 ceph-mon[102844]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:46 smithi196 ceph-mon[76349]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:46 smithi184 ceph-mon[100622]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:46 smithi184 ceph-mon[102844]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:48 smithi196 ceph-mon[76349]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[100622]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[102844]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:25:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:50 smithi184 ceph-mon[100622]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:50 smithi184 ceph-mon[102844]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:50 smithi196 ceph-mon[76349]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:51.319Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:51.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:25:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:25:51.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:25:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:25:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:25:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:52 smithi184 ceph-mon[100622]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:52 smithi184 ceph-mon[102844]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:52 smithi196 ceph-mon[76349]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:54 smithi184 ceph-mon[100622]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:54 smithi184 ceph-mon[102844]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:54 smithi196 ceph-mon[76349]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:56.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:56 smithi184 ceph-mon[100622]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:56.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:56 smithi184 ceph-mon[102844]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:56 smithi196 ceph-mon[76349]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:58 smithi196 ceph-mon[76349]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:58 smithi184 ceph-mon[100622]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:58 smithi184 ceph-mon[102844]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.804 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:00 smithi184 ceph-mon[102844]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:00 smithi184 ceph-mon[100622]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:00 smithi196 ceph-mon[76349]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:01.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:01.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:01.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:01.988 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:26:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 4h - - 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 4h - - 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 13s ago 4h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 13s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 15s ago 4h 461M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 13s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 15s ago 4h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:26:02.538 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 13s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 15s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 15s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 15s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 13s ago 4h 77.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:26:02.539 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 15s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:26:02.540 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:26:02.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:26:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:26:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:26:02.864 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:26:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:02 smithi184 ceph-mon[100622]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:02 smithi184 ceph-mon[102844]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:02 smithi196 ceph-mon[76349]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:03.143 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:26:03.143 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:26:03.143 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:26:03.143 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:26:03.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:26:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:26:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:03.465 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:03.466 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[100622]: from='client.62616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[100622]: from='client.72584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[100622]: from='client.72590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2621391527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1121817245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[102844]: from='client.62616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[102844]: from='client.72584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[102844]: from='client.72590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2621391527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1121817245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:04.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:03 smithi196 ceph-mon[76349]: from='client.62616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:03 smithi196 ceph-mon[76349]: from='client.72584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:03 smithi196 ceph-mon[76349]: from='client.72590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:04.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2621391527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:04.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1121817245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:04 smithi184 ceph-mon[100622]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:04 smithi184 ceph-mon[100622]: from='client.62640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:04 smithi184 ceph-mon[102844]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:04 smithi184 ceph-mon[102844]: from='client.62640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:04 smithi196 ceph-mon[76349]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:04 smithi196 ceph-mon[76349]: from='client.62640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:06 smithi184 ceph-mon[100622]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:06 smithi184 ceph-mon[102844]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:06 smithi196 ceph-mon[76349]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[100622]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[102844]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:08 smithi196 ceph-mon[76349]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:10 smithi196 ceph-mon[76349]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:10 smithi184 ceph-mon[100622]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:10 smithi184 ceph-mon[102844]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:11.321Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:11.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:11.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:26:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:12 smithi196 ceph-mon[76349]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:12 smithi184 ceph-mon[100622]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:12 smithi184 ceph-mon[102844]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:14 smithi196 ceph-mon[76349]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:14 smithi184 ceph-mon[100622]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:14 smithi184 ceph-mon[102844]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:16 smithi196 ceph-mon[76349]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:17.223 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:16 smithi184 ceph-mon[100622]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:17.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:16 smithi184 ceph-mon[102844]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:18 smithi196 ceph-mon[76349]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:18 smithi184 ceph-mon[100622]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:18 smithi184 ceph-mon[102844]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:21.123 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:20 smithi196 ceph-mon[76349]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:21.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:20 smithi184 ceph-mon[100622]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:20 smithi184 ceph-mon[102844]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:21.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:21.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:21.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T00:26:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:22 smithi196 ceph-mon[76349]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:22 smithi184 ceph-mon[100622]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:22 smithi184 ceph-mon[102844]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:24 smithi196 ceph-mon[76349]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:24 smithi184 ceph-mon[100622]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:24 smithi184 ceph-mon[102844]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:26 smithi196 ceph-mon[76349]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:26 smithi184 ceph-mon[100622]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:26 smithi184 ceph-mon[102844]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:28 smithi196 ceph-mon[76349]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[100622]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[102844]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:31.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:31.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:31.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:32 smithi184 ceph-mon[100622]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:32 smithi184 ceph-mon[102844]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:26:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:32 smithi196 ceph-mon[76349]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:33.763 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:26:34.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:34 smithi184 ceph-mon[102844]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:34 smithi184 ceph-mon[102844]: from='client.62652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:34 smithi184 ceph-mon[100622]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:34 smithi184 ceph-mon[100622]: from='client.62652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:34.302 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:26:34.302 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-21T00:26:34.302 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 4h - - 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 5s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 4h 461M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 5s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 5s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:26:34.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 5s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 5s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 5s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 5s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 5s ago 4h 72.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:26:34.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 5s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:26:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:34 smithi196 ceph-mon[76349]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:34 smithi196 ceph-mon[76349]: from='client.62652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:26:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:26:34.618 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:26:34.886 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:26:35.195 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:26:35.195 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:35.196 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:35.196 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:35.196 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:35.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[100622]: from='client.72620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[100622]: from='client.62664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/300128206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:35.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[102844]: from='client.72620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[102844]: from='client.62664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/300128206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:35 smithi196 ceph-mon[76349]: from='client.72620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:35 smithi196 ceph-mon[76349]: from='client.62664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/300128206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[100622]: from='client.72638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[100622]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3013882501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[102844]: from='client.72638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[102844]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3013882501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:36 smithi196 ceph-mon[76349]: from='client.72638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:36 smithi196 ceph-mon[76349]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3013882501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:38 smithi184 ceph-mon[100622]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:38 smithi184 ceph-mon[102844]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:38 smithi196 ceph-mon[76349]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:40 smithi196 ceph-mon[76349]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:40 smithi184 ceph-mon[100622]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:40 smithi184 ceph-mon[102844]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:41.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:41.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:41.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:42 smithi184 ceph-mon[100622]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:42 smithi184 ceph-mon[102844]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:26:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:42 smithi196 ceph-mon[76349]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:44 smithi184 ceph-mon[100622]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:44 smithi184 ceph-mon[102844]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:44 smithi196 ceph-mon[76349]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:46 smithi196 ceph-mon[76349]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:46 smithi184 ceph-mon[100622]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:46 smithi184 ceph-mon[102844]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[100622]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[102844]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:48 smithi196 ceph-mon[76349]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:50 smithi196 ceph-mon[76349]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[100622]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[102844]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:26:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:51.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:51.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:26:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:26:51.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:26:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:52 smithi184 ceph-mon[100622]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:52 smithi184 ceph-mon[102844]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:26:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:26:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:52 smithi196 ceph-mon[76349]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:54 smithi184 ceph-mon[100622]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:54 smithi184 ceph-mon[102844]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:54 smithi196 ceph-mon[76349]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:56 smithi184 ceph-mon[100622]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:56 smithi184 ceph-mon[102844]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:56 smithi196 ceph-mon[76349]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:58.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:58 smithi184 ceph-mon[100622]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:58.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:58 smithi184 ceph-mon[102844]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:58 smithi196 ceph-mon[76349]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:00 smithi184 ceph-mon[102844]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:00 smithi184 ceph-mon[100622]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:00 smithi196 ceph-mon[76349]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:01.325Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:01.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:01.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:27:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:02 smithi196 ceph-mon[76349]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:02 smithi184 ceph-mon[100622]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:02 smithi184 ceph-mon[102844]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:04 smithi184 ceph-mon[100622]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:04 smithi184 ceph-mon[102844]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:04 smithi196 ceph-mon[76349]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 4h 461M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 4h 315M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:27:06.019 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 73.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:27:06.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:27:06.329 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:27:06.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:27:06.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:27:06.331 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:27:06.599 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:27:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:27:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:27:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:27:06.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:27:06.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[100622]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:06.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[100622]: from='client.72650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[100622]: from='client.72656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2300917775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:06.600 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[102844]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:06.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[102844]: from='client.72650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[102844]: from='client.72656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2300917775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:06 smithi196 ceph-mon[76349]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:06 smithi196 ceph-mon[76349]: from='client.72650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:06 smithi196 ceph-mon[76349]: from='client.72656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2300917775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:06.909 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:27:06.909 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:06.909 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[102844]: from='client.72662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[102844]: from='client.72674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/848623300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[100622]: from='client.72662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[100622]: from='client.72674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/848623300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:07 smithi196 ceph-mon[76349]: from='client.72662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:07 smithi196 ceph-mon[76349]: from='client.72674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/848623300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:08 smithi196 ceph-mon[76349]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[100622]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[102844]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[100622]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[102844]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:10 smithi196 ceph-mon[76349]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:11.326Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:11.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:11.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:12 smithi184 ceph-mon[100622]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:12 smithi184 ceph-mon[102844]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:27:12.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:12 smithi196 ceph-mon[76349]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:14 smithi184 ceph-mon[100622]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:14 smithi184 ceph-mon[102844]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:14 smithi196 ceph-mon[76349]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:16 smithi184 ceph-mon[100622]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:16 smithi184 ceph-mon[102844]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:16 smithi196 ceph-mon[76349]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:18 smithi184 ceph-mon[100622]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:18 smithi184 ceph-mon[102844]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:18 smithi196 ceph-mon[76349]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:20 smithi184 ceph-mon[100622]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:20 smithi184 ceph-mon[102844]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:20.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:20 smithi196 ceph-mon[76349]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:20.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:21.327Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:21.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:21.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:27:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:22 smithi184 ceph-mon[100622]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:22 smithi184 ceph-mon[102844]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:22 smithi196 ceph-mon[76349]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:24 smithi184 ceph-mon[100622]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:24 smithi184 ceph-mon[102844]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:24 smithi196 ceph-mon[76349]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:26 smithi184 ceph-mon[100622]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:26 smithi184 ceph-mon[102844]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:26 smithi196 ceph-mon[76349]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[100622]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[102844]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:28 smithi196 ceph-mon[76349]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:29.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:30 smithi184 ceph-mon[100622]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:30 smithi184 ceph-mon[102844]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:30 smithi196 ceph-mon[76349]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:31.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:31.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:31.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:27:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:32 smithi184 ceph-mon[102844]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:32 smithi184 ceph-mon[100622]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:32 smithi196 ceph-mon[76349]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:34 smithi184 ceph-mon[100622]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:34 smithi184 ceph-mon[102844]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:34 smithi196 ceph-mon[76349]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:36 smithi184 ceph-mon[100622]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:36 smithi184 ceph-mon[102844]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:36 smithi196 ceph-mon[76349]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:37.205 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:27:37.731 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:27:37.731 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 4h - - 2024-09-21T00:27:37.731 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 4h - - 2024-09-21T00:27:37.731 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 10s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 10s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 10s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 10s ago 4h 461M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 8s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 10s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 10s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 10s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 10s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:27:37.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 8s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 8s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 8s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 8s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 74.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:27:37.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 8s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:27:38.041 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:27:38.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:27:38.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:27:38.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:27:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:27:38.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:27:38.619 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:27:38.619 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:38.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:38.619 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:38.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: from='client.62730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1700049279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2942162090' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: from='client.62730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1700049279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:39.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2942162090' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: from='client.62730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1700049279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2942162090' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:39.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:39 smithi184 ceph-mon[100622]: from='client.62748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:39.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:39 smithi184 ceph-mon[102844]: from='client.62748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:39 smithi196 ceph-mon[76349]: from='client.62748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:40 smithi196 ceph-mon[76349]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:40 smithi184 ceph-mon[100622]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:40 smithi184 ceph-mon[102844]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:41.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:41.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:41.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:27:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:42 smithi184 ceph-mon[102844]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:42 smithi184 ceph-mon[100622]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:42 smithi196 ceph-mon[76349]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:44 smithi184 ceph-mon[102844]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:44 smithi184 ceph-mon[100622]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:44 smithi196 ceph-mon[76349]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:46 smithi184 ceph-mon[100622]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:46 smithi184 ceph-mon[102844]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:46 smithi196 ceph-mon[76349]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:48 smithi196 ceph-mon[76349]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[102844]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[100622]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[100622]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[102844]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:50 smithi196 ceph-mon[76349]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:27:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:51.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:51.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:27:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:27:51.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:27:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:27:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T00:27:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:52 smithi184 ceph-mon[100622]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:52 smithi184 ceph-mon[102844]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:52 smithi196 ceph-mon[76349]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:54 smithi184 ceph-mon[100622]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:54 smithi184 ceph-mon[102844]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:54 smithi196 ceph-mon[76349]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:56 smithi184 ceph-mon[100622]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:56 smithi184 ceph-mon[102844]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:56 smithi196 ceph-mon[76349]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:58 smithi184 ceph-mon[100622]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:58 smithi184 ceph-mon[102844]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:58 smithi196 ceph-mon[76349]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:00 smithi184 ceph-mon[100622]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:00 smithi184 ceph-mon[102844]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:00 smithi196 ceph-mon[76349]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:01.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:01.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:01.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:28:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:02 smithi196 ceph-mon[76349]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:02 smithi184 ceph-mon[100622]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:02 smithi184 ceph-mon[102844]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:04 smithi184 ceph-mon[100622]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:04 smithi184 ceph-mon[102844]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:04 smithi196 ceph-mon[76349]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:06 smithi184 ceph-mon[100622]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:06 smithi184 ceph-mon[102844]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:06 smithi196 ceph-mon[76349]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[100622]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[102844]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:08 smithi196 ceph-mon[76349]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:08.915 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:28:09.444 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:28:09.444 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-21T00:28:09.444 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-21T00:28:09.444 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 1s ago 4h 27.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 1s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 1s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 1s ago 4h 462M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 1s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 1s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 1s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 1s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 1s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 1s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:28:09.445 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 74.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 1s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:28:09.446 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:28:09.760 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:28:09.760 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:28:09.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:28:09.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:28:10.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:28:10.030 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:28:10.030 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:28:10.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:28:10.339 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:28:10.339 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:10.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:10.339 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:10.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.72722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.62766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.62772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/900659353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/97046218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.72722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.62766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.62772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/900659353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/97046218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.72722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.62766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.62772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/900659353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/97046218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:11.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:11.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:11.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:11 smithi196 ceph-mon[76349]: from='client.72746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:11 smithi184 ceph-mon[100622]: from='client.72746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:11 smithi184 ceph-mon[102844]: from='client.72746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:28:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:12 smithi196 ceph-mon[76349]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:12 smithi184 ceph-mon[100622]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:12 smithi184 ceph-mon[102844]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:28:13.812690535Z level=info msg="Completed cleanup jobs" duration=12.573881ms 2024-09-21T00:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:14 smithi196 ceph-mon[76349]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:14 smithi184 ceph-mon[102844]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:14 smithi184 ceph-mon[100622]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:16 smithi196 ceph-mon[76349]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:16 smithi184 ceph-mon[100622]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:16 smithi184 ceph-mon[102844]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:18 smithi196 ceph-mon[76349]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:18 smithi184 ceph-mon[102844]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:18 smithi184 ceph-mon[100622]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:20.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:20 smithi184 ceph-mon[102844]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:20.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:20.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:20 smithi184 ceph-mon[100622]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:20.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:20 smithi196 ceph-mon[76349]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:21.608 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:21.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:21.608 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:21.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:21.608 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:21.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:22.205 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:28:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:22 smithi184 ceph-mon[100622]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:22 smithi184 ceph-mon[102844]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:22 smithi196 ceph-mon[76349]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:24 smithi196 ceph-mon[76349]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:24 smithi184 ceph-mon[100622]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:24 smithi184 ceph-mon[102844]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:26 smithi196 ceph-mon[76349]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:26 smithi184 ceph-mon[100622]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:26 smithi184 ceph-mon[102844]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:28 smithi196 ceph-mon[76349]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[100622]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[102844]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[100622]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[102844]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:30 smithi196 ceph-mon[76349]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:31.332Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:31.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:31.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:28:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:32 smithi184 ceph-mon[100622]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:32 smithi184 ceph-mon[102844]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:32 smithi196 ceph-mon[76349]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:34 smithi196 ceph-mon[76349]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:34 smithi184 ceph-mon[100622]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:34 smithi184 ceph-mon[102844]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:36 smithi184 ceph-mon[100622]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:36 smithi184 ceph-mon[102844]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:36 smithi196 ceph-mon[76349]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:38 smithi184 ceph-mon[100622]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:28:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:38 smithi184 ceph-mon[102844]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:28:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:38 smithi196 ceph-mon[76349]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:28:40.634 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:28:40.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:40 smithi184 ceph-mon[100622]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:40.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:40.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:40 smithi184 ceph-mon[102844]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:40.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:40 smithi196 ceph-mon[76349]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:41.185 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 4h - - 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 13s ago 4h 27.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 11s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 13s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 11s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 13s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 13s ago 4h 463M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 11s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 13s ago 4h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 13s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 11s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:28:41.186 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 13s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 13s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 13s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 13s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 11s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 11s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 11s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 11s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 11s ago 4h 76.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 13s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:28:41.187 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 11s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:28:41.504 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:28:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:28:41.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:28:41.506 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:28:41.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:41.332Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:41.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:41.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:41.781 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:28:41.782 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:28:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-mon[100622]: from='client.72758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1252027520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-mon[102844]: from='client.72758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1252027520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:42.113 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:28:42.113 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:42.114 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:42.114 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:42.114 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:41 smithi196 ceph-mon[76349]: from='client.72758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1252027520' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:28:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:42 smithi196 ceph-mon[76349]: from='client.62802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:42 smithi196 ceph-mon[76349]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:42 smithi196 ceph-mon[76349]: from='client.62808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:42 smithi196 ceph-mon[76349]: from='client.62817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1805885084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[100622]: from='client.62802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[100622]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[100622]: from='client.62808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[100622]: from='client.62817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1805885084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[102844]: from='client.62802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[102844]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[102844]: from='client.62808 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[102844]: from='client.62817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1805885084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:44 smithi184 ceph-mon[102844]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:44 smithi184 ceph-mon[100622]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:44 smithi196 ceph-mon[76349]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:46 smithi184 ceph-mon[100622]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:46 smithi184 ceph-mon[102844]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:46 smithi196 ceph-mon[76349]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[102844]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[100622]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:48 smithi196 ceph-mon[76349]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:50 smithi196 ceph-mon[76349]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[100622]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[102844]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:28:51.638 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:51.333Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:51.638 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:51.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:28:51.638 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:28:51.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:28:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:28:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:28:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:52 smithi196 ceph-mon[76349]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:52 smithi184 ceph-mon[100622]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:52 smithi184 ceph-mon[102844]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:54 smithi196 ceph-mon[76349]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:54 smithi184 ceph-mon[100622]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:54 smithi184 ceph-mon[102844]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:56 smithi196 ceph-mon[76349]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:56 smithi184 ceph-mon[100622]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:56 smithi184 ceph-mon[102844]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:58 smithi196 ceph-mon[76349]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:58 smithi184 ceph-mon[100622]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:58 smithi184 ceph-mon[102844]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:00 smithi196 ceph-mon[76349]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:00 smithi184 ceph-mon[100622]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:00 smithi184 ceph-mon[102844]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:01.709 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:01.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:01.709 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:01.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:01.709 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:01.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:29:03.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:02 smithi184 ceph-mon[100622]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:03.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:02 smithi184 ceph-mon[102844]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:03.079 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:02 smithi196 ceph-mon[76349]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:05.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:04 smithi196 ceph-mon[76349]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:04 smithi184 ceph-mon[100622]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:05.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:04 smithi184 ceph-mon[102844]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:06 smithi196 ceph-mon[76349]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:06 smithi184 ceph-mon[100622]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:06 smithi184 ceph-mon[102844]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:08 smithi196 ceph-mon[76349]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[100622]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[102844]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:09.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:10 smithi196 ceph-mon[76349]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[100622]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[102844]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:11.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:11.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:11.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:29:12.401 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:29:12.928 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:29:12.928 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-21T00:29:12.928 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-21T00:29:12.928 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 4s ago 4h 27.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:29:12.928 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 4s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 4s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 4s ago 4h 465M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 4s ago 4h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 4s ago 4h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 4s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 4s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 4s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 4s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:29:12.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 77.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 4s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:29:12.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:29:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:29:13.240 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:29:13.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:12 smithi184 ceph-mon[100622]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:13.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:12 smithi184 ceph-mon[102844]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:12 smithi196 ceph-mon[76349]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:29:13.507 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:29:13.508 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:29:13.508 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:29:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:29:13.819 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:29:13.819 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:13.819 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:14.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[100622]: from='client.62829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[100622]: from='client.72800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/430007267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3865970764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[102844]: from='client.62829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[102844]: from='client.72800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/430007267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:14.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3865970764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:13 smithi196 ceph-mon[76349]: from='client.62829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:13 smithi196 ceph-mon[76349]: from='client.72800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/430007267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3865970764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[100622]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[100622]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[100622]: from='client.72818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[102844]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[102844]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[102844]: from='client.72818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:14 smithi196 ceph-mon[76349]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:14 smithi196 ceph-mon[76349]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:14 smithi196 ceph-mon[76349]: from='client.72818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:15 smithi196 ceph-mon[76349]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:15 smithi184 ceph-mon[100622]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:15 smithi184 ceph-mon[102844]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:18 smithi196 ceph-mon[76349]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:18 smithi184 ceph-mon[100622]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:18 smithi184 ceph-mon[102844]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:20 smithi196 ceph-mon[76349]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:20 smithi184 ceph-mon[100622]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:20 smithi184 ceph-mon[102844]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:21.336Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:21.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:21.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:22 smithi184 ceph-mon[100622]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:29:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:22 smithi184 ceph-mon[102844]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:22 smithi196 ceph-mon[76349]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:24 smithi196 ceph-mon[76349]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:24.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:24 smithi184 ceph-mon[100622]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:24.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:24 smithi184 ceph-mon[102844]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:26 smithi196 ceph-mon[76349]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:26 smithi184 ceph-mon[100622]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:26 smithi184 ceph-mon[102844]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:28 smithi196 ceph-mon[76349]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:28.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:28 smithi184 ceph-mon[100622]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:28.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:28 smithi184 ceph-mon[102844]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[100622]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[102844]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:30 smithi196 ceph-mon[76349]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:31.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:31.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:31.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:29:32.736 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:32 smithi184 ceph-mon[100622]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:32.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:32 smithi184 ceph-mon[102844]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:32 smithi196 ceph-mon[76349]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:34 smithi184 ceph-mon[100622]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:34 smithi184 ceph-mon[102844]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:34 smithi196 ceph-mon[76349]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:36 smithi184 ceph-mon[100622]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:36 smithi184 ceph-mon[102844]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:36 smithi196 ceph-mon[76349]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:38 smithi184 ceph-mon[100622]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:38 smithi184 ceph-mon[102844]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:38 smithi196 ceph-mon[76349]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:40 smithi184 ceph-mon[100622]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:40 smithi184 ceph-mon[102844]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:40.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:40 smithi196 ceph-mon[76349]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:40.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:41.338Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:41.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:41.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:29:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:42 smithi184 ceph-mon[102844]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:42 smithi184 ceph-mon[100622]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:42 smithi196 ceph-mon[76349]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:44.137 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:29:44.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:44 smithi184 ceph-mon[100622]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:44.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:44.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:44 smithi184 ceph-mon[102844]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:44.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:29:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-21T00:29:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-21T00:29:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 16s ago 4h 27.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:29:44.686 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 16s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 16s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 16s ago 4h 465M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 16s ago 4h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 16s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 16s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 16s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 16s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:29:44.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 14s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 14s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 77.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 16s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:29:44.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 14s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:29:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:44 smithi196 ceph-mon[76349]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:29:45.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:29:45.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:29:45.030 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:29:45.315 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:29:45.316 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:29:45.629 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:29:45.629 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:45.629 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:45.630 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:45.630 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:45.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[100622]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[100622]: from='client.62871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[100622]: from='client.72842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.630 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3706727694' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:45.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[102844]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[102844]: from='client.62871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[102844]: from='client.72842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3706727694' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:45 smithi196 ceph-mon[76349]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:45 smithi196 ceph-mon[76349]: from='client.62871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:45 smithi196 ceph-mon[76349]: from='client.72842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3706727694' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:46 smithi196 ceph-mon[76349]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:46 smithi196 ceph-mon[76349]: from='client.62889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4193349856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[100622]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[100622]: from='client.62889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4193349856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[102844]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[102844]: from='client.62889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4193349856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:48 smithi196 ceph-mon[76349]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:48 smithi184 ceph-mon[100622]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:48 smithi184 ceph-mon[102844]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[102844]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[100622]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:50 smithi196 ceph-mon[76349]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:29:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:51.338Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:51.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:29:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:29:51.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:29:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:29:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:29:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:52 smithi184 ceph-mon[102844]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:52 smithi184 ceph-mon[100622]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:52 smithi196 ceph-mon[76349]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:54 smithi184 ceph-mon[102844]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:54 smithi184 ceph-mon[100622]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:54 smithi196 ceph-mon[76349]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:56 smithi184 ceph-mon[100622]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:57.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:56 smithi184 ceph-mon[102844]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:56 smithi196 ceph-mon[76349]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:58 smithi184 ceph-mon[100622]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:58 smithi184 ceph-mon[102844]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:58 smithi196 ceph-mon[76349]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[102844]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[100622]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:00 smithi196 ceph-mon[76349]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:01.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:01.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:01.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:01.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:30:02.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:02 smithi196 ceph-mon[76349]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:02 smithi184 ceph-mon[102844]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:02 smithi184 ceph-mon[100622]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:04 smithi196 ceph-mon[76349]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:04.945 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:04 smithi184 ceph-mon[100622]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:04.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:04 smithi184 ceph-mon[102844]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:07.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:06 smithi196 ceph-mon[76349]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:06 smithi184 ceph-mon[100622]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:06 smithi184 ceph-mon[102844]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:08 smithi184 ceph-mon[100622]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:08 smithi184 ceph-mon[102844]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:08 smithi196 ceph-mon[76349]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[102844]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[100622]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:10 smithi196 ceph-mon[76349]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:11.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:11.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:11.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:30:12.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:12 smithi184 ceph-mon[100622]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:12.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:12 smithi184 ceph-mon[102844]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:12 smithi196 ceph-mon[76349]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:14 smithi184 ceph-mon[100622]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:14 smithi184 ceph-mon[102844]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:14 smithi196 ceph-mon[76349]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:15.927 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:30:16.455 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:30:16.455 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 4h - - 2024-09-21T00:30:16.455 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-21T00:30:16.455 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:30:16.455 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 4h 466M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 4h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:30:16.456 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 4h 77.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:30:16.457 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:30:16.764 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:30:16.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:30:16.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:30:16.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:16.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:30:16.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:30:16.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:16.766 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:30:16.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:30:16.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:30:16.766 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:30:17.035 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:30:17.036 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:30:17.037 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:30:17.037 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:30:17.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:16 smithi184 ceph-mon[100622]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:17.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1568307577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:17.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:16 smithi184 ceph-mon[102844]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:17.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1568307577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:16 smithi196 ceph-mon[76349]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1568307577' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:17.345 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:17.345 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:17.346 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:17.346 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:17.346 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[100622]: from='client.62901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[100622]: from='client.72872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[100622]: from='client.62913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2796793337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[102844]: from='client.62901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[102844]: from='client.72872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[102844]: from='client.62913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2796793337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:17 smithi196 ceph-mon[76349]: from='client.62901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:17 smithi196 ceph-mon[76349]: from='client.72872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:17 smithi196 ceph-mon[76349]: from='client.62913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2796793337' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:18 smithi196 ceph-mon[76349]: from='client.62925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:18 smithi196 ceph-mon[76349]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:18 smithi184 ceph-mon[102844]: from='client.62925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:18 smithi184 ceph-mon[102844]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:18 smithi184 ceph-mon[100622]: from='client.62925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:18 smithi184 ceph-mon[100622]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:20 smithi196 ceph-mon[76349]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:20 smithi184 ceph-mon[100622]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:20 smithi184 ceph-mon[102844]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:21.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:21.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:21.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:30:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:22 smithi196 ceph-mon[76349]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:22 smithi184 ceph-mon[100622]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:22 smithi184 ceph-mon[102844]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:24 smithi196 ceph-mon[76349]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:24 smithi184 ceph-mon[100622]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:24 smithi184 ceph-mon[102844]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:26 smithi196 ceph-mon[76349]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:27.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:26 smithi184 ceph-mon[100622]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:27.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:26 smithi184 ceph-mon[102844]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:28 smithi196 ceph-mon[76349]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:28 smithi184 ceph-mon[100622]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:28 smithi184 ceph-mon[102844]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[100622]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[102844]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:30 smithi196 ceph-mon[76349]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:31.341Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:31.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:31.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:32.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:31 smithi196 ceph-mon[76349]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:31 smithi184 ceph-mon[100622]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:31 smithi184 ceph-mon[102844]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:34 smithi196 ceph-mon[76349]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:34 smithi184 ceph-mon[100622]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:34 smithi184 ceph-mon[102844]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:36 smithi196 ceph-mon[76349]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:36 smithi184 ceph-mon[100622]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:36 smithi184 ceph-mon[102844]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:38 smithi196 ceph-mon[76349]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:38 smithi184 ceph-mon[100622]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:38 smithi184 ceph-mon[102844]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:40 smithi196 ceph-mon[76349]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:40 smithi184 ceph-mon[100622]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:40 smithi184 ceph-mon[102844]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:41.341Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:41.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:41.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:42 smithi184 ceph-mon[100622]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:42 smithi184 ceph-mon[102844]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:30:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:42 smithi196 ceph-mon[76349]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:44 smithi196 ceph-mon[76349]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:44 smithi184 ceph-mon[100622]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:44 smithi184 ceph-mon[102844]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:46.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:46 smithi184 ceph-mon[100622]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:46.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:46 smithi184 ceph-mon[102844]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:46 smithi196 ceph-mon[76349]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:47.637 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 4h - - 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 27.9M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 4h 466M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:30:48.169 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 4h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 77.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:30:48.170 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:30:48.171 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:48 smithi196 ceph-mon[76349]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:48 smithi196 ceph-mon[76349]: from='client.62937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:48 smithi196 ceph-mon[76349]: from='client.72908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:30:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:30:48.479 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:30:48.479 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[100622]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:48.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[100622]: from='client.62937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.480 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[100622]: from='client.72908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[102844]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:48.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[102844]: from='client.62937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.480 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:48 smithi184 ceph-mon[102844]: from='client.72908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:30:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:30:48.748 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:30:49.070 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:49.070 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:49.070 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:49.070 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:49.070 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2051397288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='client.62961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3214773089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2051397288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='client.62961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.301 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3214773089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2051397288' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='client.62961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3214773089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:50 smithi184 ceph-mon[100622]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:50 smithi184 ceph-mon[102844]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:50 smithi196 ceph-mon[76349]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:51.626 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:51.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:51.626 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:51.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:30:51.626 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:30:51.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.004 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:30:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:30:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:30:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:52 smithi184 ceph-mon[100622]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:52 smithi184 ceph-mon[102844]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:52 smithi196 ceph-mon[76349]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:54 smithi184 ceph-mon[100622]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:54 smithi184 ceph-mon[102844]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:54 smithi196 ceph-mon[76349]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:56 smithi184 ceph-mon[100622]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:56 smithi184 ceph-mon[102844]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:56 smithi196 ceph-mon[76349]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:58 smithi184 ceph-mon[100622]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:58 smithi184 ceph-mon[102844]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:58 smithi196 ceph-mon[76349]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:00 smithi184 ceph-mon[100622]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:00 smithi184 ceph-mon[102844]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:00 smithi196 ceph-mon[76349]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:01.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:01.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:01.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:31:02.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:02 smithi196 ceph-mon[76349]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:02 smithi184 ceph-mon[100622]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:02 smithi184 ceph-mon[102844]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:04 smithi184 ceph-mon[100622]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:04 smithi184 ceph-mon[102844]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:04 smithi196 ceph-mon[76349]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:06.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:06 smithi184 ceph-mon[100622]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:06.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:06 smithi184 ceph-mon[102844]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:06 smithi196 ceph-mon[76349]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:08.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:08 smithi196 ceph-mon[76349]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:08 smithi184 ceph-mon[100622]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:08 smithi184 ceph-mon[102844]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[100622]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[102844]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:10 smithi196 ceph-mon[76349]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:11.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:11.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:11.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:11.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:11.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:11.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:31:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:12 smithi184 ceph-mon[100622]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:12 smithi184 ceph-mon[102844]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:12 smithi196 ceph-mon[76349]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:14 smithi184 ceph-mon[100622]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:14 smithi184 ceph-mon[102844]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:14 smithi196 ceph-mon[76349]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:16 smithi184 ceph-mon[100622]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:16 smithi184 ceph-mon[102844]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:16 smithi196 ceph-mon[76349]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:18 smithi184 ceph-mon[100622]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:18 smithi184 ceph-mon[102844]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:18 smithi196 ceph-mon[76349]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:19.364 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 4h - - 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 10s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 9s ago 4h 91.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 10s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 10s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:31:19.893 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 10s ago 4h 468M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 9s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 10s ago 4h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 10s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 9s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 10s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 10s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 9s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 9s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:31:19.894 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 9s ago 4h 77.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:31:19.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:31:19.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:31:19.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:31:19.895 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 9s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:31:20.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:31:20.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:31:20.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:31:20.204 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:31:20.470 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:31:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:31:20.471 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:31:20.781 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:31:20.781 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:20.781 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:20.782 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:20.782 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.62973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.72944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3565112025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2585904263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.62973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.72944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3565112025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2585904263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.62973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.72944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3565112025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2585904263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:21.792 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:21.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:21.792 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:21.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:21.792 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:21.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:22.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:21 smithi184 ceph-mon[102844]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:21 smithi184 ceph-mon[100622]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:31:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:21 smithi196 ceph-mon[76349]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:22 smithi184 ceph-mon[100622]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:22 smithi184 ceph-mon[102844]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:22 smithi196 ceph-mon[76349]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:24 smithi184 ceph-mon[100622]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:24 smithi184 ceph-mon[102844]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:24 smithi196 ceph-mon[76349]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:27.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:26 smithi196 ceph-mon[76349]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:27.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:26 smithi184 ceph-mon[100622]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:27.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:26 smithi184 ceph-mon[102844]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:28 smithi184 ceph-mon[100622]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:28 smithi184 ceph-mon[102844]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:28 smithi196 ceph-mon[76349]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.131 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:30 smithi196 ceph-mon[76349]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:30 smithi184 ceph-mon[102844]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:30 smithi184 ceph-mon[100622]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:31.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:31.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:31.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:32 smithi184 ceph-mon[100622]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:32 smithi184 ceph-mon[102844]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:33.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:32 smithi196 ceph-mon[76349]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:34 smithi184 ceph-mon[100622]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:34 smithi184 ceph-mon[102844]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:34 smithi196 ceph-mon[76349]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:36 smithi184 ceph-mon[100622]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:36 smithi184 ceph-mon[102844]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:36 smithi196 ceph-mon[76349]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:38 smithi184 ceph-mon[100622]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:38 smithi184 ceph-mon[102844]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:38 smithi196 ceph-mon[76349]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:40 smithi184 ceph-mon[100622]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:40 smithi184 ceph-mon[102844]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:40 smithi196 ceph-mon[76349]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:41.345Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:41.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:41.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:31:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:42 smithi196 ceph-mon[76349]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:42 smithi184 ceph-mon[100622]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:42 smithi184 ceph-mon[102844]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:43 smithi184 ceph-mon[100622]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:43 smithi184 ceph-mon[102844]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:43 smithi196 ceph-mon[76349]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:45 smithi184 ceph-mon[100622]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:45 smithi184 ceph-mon[102844]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:45 smithi196 ceph-mon[76349]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:48.202 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:48 smithi184 ceph-mon[100622]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:48.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:48 smithi184 ceph-mon[102844]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:48.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:48 smithi196 ceph-mon[76349]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:50.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[100622]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:50.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[102844]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:50 smithi196 ceph-mon[76349]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:51.078 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:31:51.607 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:31:51.607 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 4h - - 2024-09-21T00:31:51.607 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 4h - - 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 4h 468M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 4h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:31:51.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 0s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 0s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 0s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 77.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:31:51.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:31:51.609 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:51.346Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:51.609 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:51.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:31:51.609 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:31:51.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:31:51.914 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:31:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:31:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:31:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:31:51.916 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:31:52.187 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:31:52.187 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:31:52.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:31:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='client.63009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='client.63015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='client.72986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/466972370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='client.63009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='client.63015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='client.72986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/466972370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:31:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='client.63009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='client.63015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='client.72986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/466972370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:52.496 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:31:52.496 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:52.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:52.496 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:52.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:53 smithi184 ceph-mon[100622]: from='client.63033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/174138473' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:53 smithi184 ceph-mon[102844]: from='client.63033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/174138473' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:53 smithi196 ceph-mon[76349]: from='client.63033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/174138473' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:54.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:54 smithi184 ceph-mon[102844]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:54 smithi184 ceph-mon[100622]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:54 smithi196 ceph-mon[76349]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:56 smithi184 ceph-mon[102844]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:56 smithi184 ceph-mon[100622]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:56 smithi196 ceph-mon[76349]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:58 smithi184 ceph-mon[102844]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:58 smithi184 ceph-mon[100622]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:58 smithi196 ceph-mon[76349]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:00 smithi196 ceph-mon[76349]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:00 smithi184 ceph-mon[102844]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:00 smithi184 ceph-mon[100622]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:01.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:01.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:01.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:02 smithi184 ceph-mon[100622]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:02 smithi184 ceph-mon[102844]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:32:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:02 smithi196 ceph-mon[76349]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:04 smithi184 ceph-mon[100622]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:04 smithi184 ceph-mon[102844]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:04 smithi196 ceph-mon[76349]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:06 smithi184 ceph-mon[100622]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:06 smithi184 ceph-mon[102844]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:06 smithi196 ceph-mon[76349]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:08.434 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:08 smithi184 ceph-mon[102844]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:08.434 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:08 smithi184 ceph-mon[100622]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:08 smithi196 ceph-mon[76349]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[100622]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[102844]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:10 smithi196 ceph-mon[76349]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:11.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:11.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:11.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[100622]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[102844]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:12 smithi196 ceph-mon[76349]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:14 smithi196 ceph-mon[76349]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:14 smithi184 ceph-mon[100622]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:14 smithi184 ceph-mon[102844]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:16 smithi196 ceph-mon[76349]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:16 smithi184 ceph-mon[100622]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:16 smithi184 ceph-mon[102844]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:18 smithi184 ceph-mon[100622]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:18 smithi184 ceph-mon[102844]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:18 smithi196 ceph-mon[76349]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:20 smithi184 ceph-mon[102844]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:20 smithi184 ceph-mon[100622]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:20 smithi196 ceph-mon[76349]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:21.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:21.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:21.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:22 smithi184 ceph-mon[102844]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:22 smithi184 ceph-mon[100622]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:32:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:22 smithi196 ceph-mon[76349]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:22.789 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 4h - - 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 13s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 13s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 13s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 13s ago 4h 468M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:32:23.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 13s ago 4h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 13s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 13s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 13s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 13s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 13s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:32:23.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 13s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:32:23.319 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:32:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:23 smithi184 ceph-mon[102844]: from='client.63045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:23 smithi184 ceph-mon[100622]: from='client.63045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:23.629 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:23.630 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:32:23.631 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:32:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:23 smithi196 ceph-mon[76349]: from='client.63045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:32:23.897 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:32:23.898 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:32:23.898 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:32:24.209 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:32:24.209 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:24.209 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:24.209 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:24.210 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[102844]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[102844]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[102844]: from='client.73022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1066692614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[102844]: from='client.63069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[100622]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[100622]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[100622]: from='client.73022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1066692614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:24.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:24 smithi184 ceph-mon[100622]: from='client.63069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:24 smithi196 ceph-mon[76349]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:24 smithi196 ceph-mon[76349]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:24 smithi196 ceph-mon[76349]: from='client.73022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1066692614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:24 smithi196 ceph-mon[76349]: from='client.63069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3487115469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3487115469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3487115469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:26 smithi184 ceph-mon[100622]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:26 smithi184 ceph-mon[102844]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:26 smithi196 ceph-mon[76349]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:28.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:28 smithi184 ceph-mon[100622]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:28.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:28 smithi184 ceph-mon[102844]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:28 smithi196 ceph-mon[76349]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:29.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:29.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:30 smithi196 ceph-mon[76349]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:30.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[100622]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:30.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[102844]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:30.949 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:31.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:31.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:31.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[100622]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:32:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[102844]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:32 smithi196 ceph-mon[76349]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:34 smithi184 ceph-mon[100622]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:34 smithi184 ceph-mon[102844]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:34 smithi196 ceph-mon[76349]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:36 smithi184 ceph-mon[100622]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:36 smithi184 ceph-mon[102844]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:36 smithi196 ceph-mon[76349]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:38 smithi184 ceph-mon[100622]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:38 smithi184 ceph-mon[102844]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:38 smithi196 ceph-mon[76349]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:40 smithi184 ceph-mon[100622]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:40 smithi184 ceph-mon[102844]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:40 smithi196 ceph-mon[76349]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:41.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:41.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:41.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:42.263 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:32:42.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:42 smithi184 ceph-mon[102844]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:42 smithi184 ceph-mon[100622]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:42 smithi196 ceph-mon[76349]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:44 smithi184 ceph-mon[100622]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:44 smithi184 ceph-mon[102844]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:44 smithi196 ceph-mon[76349]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:46 smithi184 ceph-mon[100622]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:46 smithi184 ceph-mon[102844]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:46 smithi196 ceph-mon[76349]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:48 smithi184 ceph-mon[100622]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:48 smithi184 ceph-mon[102844]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:48 smithi196 ceph-mon[76349]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[100622]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[102844]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:50 smithi196 ceph-mon[76349]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:51.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:51.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:32:51.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:32:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:32:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:32:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:52 smithi196 ceph-mon[76349]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[100622]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[102844]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:32:54.508 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:32:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:54 smithi196 ceph-mon[76349]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:54.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:54 smithi184 ceph-mon[100622]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:54.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:54 smithi184 ceph-mon[102844]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 4h - - 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 4h - - 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 5s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 3s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 5s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 3s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 5s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 5s ago 4h 469M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 3s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 5s ago 4h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 5s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:32:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 3s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 5s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 5s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 5s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 5s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 3s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 3s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 3s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 3s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:32:55.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:32:55.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:32:55.380 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:32:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:55 smithi196 ceph-mon[76349]: from='client.73046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:55 smithi196 ceph-mon[76349]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.652 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:32:55.653 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:32:55.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:55 smithi184 ceph-mon[102844]: from='client.73046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:55 smithi184 ceph-mon[102844]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:55 smithi184 ceph-mon[100622]: from='client.73046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:55 smithi184 ceph-mon[100622]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:55.961 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:32:55.962 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:55.962 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:56 smithi196 ceph-mon[76349]: from='client.73058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:56 smithi196 ceph-mon[76349]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1632367110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:56 smithi196 ceph-mon[76349]: from='client.63105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1633755378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[100622]: from='client.73058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[100622]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1632367110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[100622]: from='client.63105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1633755378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[102844]: from='client.73058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[102844]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1632367110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[102844]: from='client.63105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1633755378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:58 smithi196 ceph-mon[76349]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:58 smithi184 ceph-mon[100622]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:58 smithi184 ceph-mon[102844]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:59.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:59.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:00 smithi184 ceph-mon[100622]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:00 smithi184 ceph-mon[102844]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:00 smithi196 ceph-mon[76349]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:01.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:01.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:01.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:33:02.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:02 smithi196 ceph-mon[76349]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:02.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:02 smithi184 ceph-mon[100622]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:02.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:02 smithi184 ceph-mon[102844]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:04 smithi184 ceph-mon[100622]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:04 smithi184 ceph-mon[102844]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:04.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:04 smithi196 ceph-mon[76349]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:06 smithi184 ceph-mon[100622]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:06 smithi184 ceph-mon[102844]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:06 smithi196 ceph-mon[76349]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:08 smithi184 ceph-mon[100622]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:08 smithi184 ceph-mon[102844]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:08 smithi196 ceph-mon[76349]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:11.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:10 smithi196 ceph-mon[76349]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:11.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:11.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[100622]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[102844]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:11.624 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:11.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:11.624 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:11.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:11.624 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:11.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[100622]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[102844]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:12 smithi196 ceph-mon[76349]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:12.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:12.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:14 smithi184 ceph-mon[102844]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:14 smithi184 ceph-mon[100622]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:14 smithi196 ceph-mon[76349]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:16 smithi184 ceph-mon[102844]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:16 smithi184 ceph-mon[100622]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:16 smithi196 ceph-mon[76349]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:18 smithi184 ceph-mon[102844]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:18 smithi184 ceph-mon[100622]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:18 smithi196 ceph-mon[76349]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:20 smithi184 ceph-mon[100622]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:20 smithi184 ceph-mon[102844]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:20 smithi196 ceph-mon[76349]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:21.352Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:21.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:21.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:33:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:22 smithi184 ceph-mon[100622]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:22 smithi184 ceph-mon[102844]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:22.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:22 smithi196 ceph-mon[76349]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:24 smithi184 ceph-mon[100622]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:24 smithi184 ceph-mon[102844]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:24 smithi196 ceph-mon[76349]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:26.259 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:33:26.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:26 smithi184 ceph-mon[100622]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:26.785 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:26 smithi184 ceph-mon[102844]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 4h - - 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 4h - - 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 16s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 16s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 15s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 16s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 16s ago 4h 470M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 15s ago 4h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:33:26.786 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 16s ago 4h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 15s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 16s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 16s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 16s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 16s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 15s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 15s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 15s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 15s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 15s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:33:26.787 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 16s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:33:26.788 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 15s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:33:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:26 smithi196 ceph-mon[76349]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:27.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:33:27.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:27.095 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:33:27.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:33:27.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:33:27.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:33:27.361 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:33:27.362 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:33:27.362 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:33:27.362 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:33:27.362 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:33:27.362 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:33:27.679 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:33:27.679 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:27.679 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:27.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[100622]: from='client.63117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[100622]: from='client.73088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[100622]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/60345313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[102844]: from='client.63117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[102844]: from='client.73088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[102844]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/60345313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:27 smithi196 ceph-mon[76349]: from='client.63117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:27 smithi196 ceph-mon[76349]: from='client.73088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:27 smithi196 ceph-mon[76349]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/60345313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[100622]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[100622]: from='client.63141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3990180536' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[102844]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[102844]: from='client.63141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3990180536' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:28 smithi196 ceph-mon[76349]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:28 smithi196 ceph-mon[76349]: from='client.63141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3990180536' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:30.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:30 smithi196 ceph-mon[76349]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:30.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:30.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[100622]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[102844]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:31.430 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:31.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:31.430 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:31.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:31.431 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:31.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:32.280 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:33:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:32 smithi196 ceph-mon[76349]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[100622]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[102844]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:34 smithi184 ceph-mon[100622]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:34 smithi184 ceph-mon[102844]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:34 smithi196 ceph-mon[76349]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:36 smithi184 ceph-mon[100622]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:36 smithi184 ceph-mon[102844]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:36 smithi196 ceph-mon[76349]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:38 smithi184 ceph-mon[100622]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:38 smithi184 ceph-mon[102844]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:38 smithi196 ceph-mon[76349]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:40 smithi184 ceph-mon[100622]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:40 smithi184 ceph-mon[102844]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:40 smithi196 ceph-mon[76349]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:41.354Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:41.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:41.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:33:42.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:42 smithi196 ceph-mon[76349]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:42 smithi184 ceph-mon[100622]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:42 smithi184 ceph-mon[102844]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:44 smithi184 ceph-mon[100622]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:44 smithi184 ceph-mon[102844]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:44 smithi196 ceph-mon[76349]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:46 smithi184 ceph-mon[100622]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:46 smithi184 ceph-mon[102844]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:46 smithi196 ceph-mon[76349]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:48 smithi184 ceph-mon[100622]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:48 smithi184 ceph-mon[102844]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:48 smithi196 ceph-mon[76349]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:51.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[100622]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:51.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:51.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[102844]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:51.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:51.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:50 smithi196 ceph-mon[76349]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:51.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:51.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:51.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:51.355Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:51.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:51.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:33:51.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:33:51.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:33:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:33:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:33:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[100622]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[102844]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:52 smithi196 ceph-mon[76349]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:33:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:54 smithi184 ceph-mon[100622]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:54 smithi184 ceph-mon[102844]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:55.137 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:54 smithi196 ceph-mon[76349]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:56 smithi184 ceph-mon[100622]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:56 smithi184 ceph-mon[102844]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:56 smithi196 ceph-mon[76349]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:57.980 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 4h - - 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 4h - - 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 8s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 8s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 8s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 8s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:33:58.508 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 8s ago 4h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 8s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 8s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 8s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 8s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 8s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:33:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:33:58.824 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:33:58.824 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:33:58.824 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:33:58.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:33:58.825 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:33:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:58 smithi184 ceph-mon[100622]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:58 smithi184 ceph-mon[102844]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:33:59.097 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:33:59.098 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:33:59.098 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:33:59.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:33:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:58 smithi196 ceph-mon[76349]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:59.408 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:33:59.409 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:59.409 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:59.409 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:59.409 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='client.63153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='client.63165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2986816479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1994119862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='client.63153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='client.63165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2986816479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:00.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:33:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1994119862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='client.63153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='client.63165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2986816479' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:33:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1994119862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[100622]: from='client.73142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[100622]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[102844]: from='client.73142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[102844]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:00 smithi196 ceph-mon[76349]: from='client.73142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:00 smithi196 ceph-mon[76349]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:01.356Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:01.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:01.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:34:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:02 smithi184 ceph-mon[100622]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:02 smithi184 ceph-mon[102844]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:02 smithi196 ceph-mon[76349]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:04 smithi184 ceph-mon[100622]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:04 smithi184 ceph-mon[102844]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:04 smithi196 ceph-mon[76349]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:06 smithi184 ceph-mon[100622]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:34:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:06 smithi184 ceph-mon[102844]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:34:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:06 smithi196 ceph-mon[76349]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:34:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:08 smithi184 ceph-mon[100622]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:34:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:08 smithi184 ceph-mon[102844]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:34:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:08 smithi196 ceph-mon[76349]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:34:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:10 smithi196 ceph-mon[76349]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[100622]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:11.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:11.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[102844]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:11.357Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:11.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:11.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:12.264 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:34:13.126 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[100622]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[102844]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:12 smithi196 ceph-mon[76349]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:34:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:14 smithi184 ceph-mon[100622]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:14 smithi184 ceph-mon[102844]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:14 smithi196 ceph-mon[76349]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:16 smithi184 ceph-mon[100622]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-21T00:34:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:16 smithi184 ceph-mon[102844]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-21T00:34:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:16 smithi196 ceph-mon[76349]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-09-21T00:34:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:18 smithi184 ceph-mon[100622]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:18 smithi184 ceph-mon[102844]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:34:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:18 smithi196 ceph-mon[76349]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:34:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:20 smithi184 ceph-mon[100622]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:34:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:20 smithi184 ceph-mon[102844]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:34:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:20 smithi196 ceph-mon[76349]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:34:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:21.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:21.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:21.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:34:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:22 smithi184 ceph-mon[100622]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T00:34:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:22 smithi184 ceph-mon[102844]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T00:34:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:22 smithi196 ceph-mon[76349]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T00:34:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:24 smithi184 ceph-mon[100622]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:34:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:24 smithi184 ceph-mon[102844]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:34:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:24 smithi196 ceph-mon[76349]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T00:34:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:26 smithi184 ceph-mon[100622]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:34:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:26 smithi184 ceph-mon[102844]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:34:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:26 smithi196 ceph-mon[76349]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:34:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:28 smithi184 ceph-mon[100622]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:34:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:28 smithi184 ceph-mon[102844]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:34:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:28 smithi196 ceph-mon[76349]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:34:29.706 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:34:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:30.234 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:34:30.234 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 4h - - 2024-09-21T00:34:30.234 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 4h - - 2024-09-21T00:34:30.234 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:34:30.234 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 18s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 18s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 18s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 4h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 4h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 4h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 18s ago 4h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 18s ago 4h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 18s ago 4h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:34:30.235 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 18s ago 4h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:34:30.236 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 18s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:34:30.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:34:30.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:34:30.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 4h 89.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:34:30.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 18s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:34:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:30.552 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:34:30.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:34:30.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:34:30.554 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:34:30.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:34:30.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:34:30.554 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:34:30.822 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:34:30.822 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:34:30.822 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:34:30.822 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:34:30.823 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:34:31.136 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:34:31.136 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:31.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:31.136 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:31.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:31.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T00:34:31.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='client.63189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:31.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:31.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4141720966' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='client.63189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4141720966' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='client.63189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4141720966' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:31.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:31.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:31.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:32.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[100622]: from='client.73160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[100622]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[100622]: from='client.63213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1553262983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:32.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[102844]: from='client.73160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[102844]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[102844]: from='client.63213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1553262983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:32.148 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:34:32.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:31 smithi196 ceph-mon[76349]: from='client.73160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:31 smithi196 ceph-mon[76349]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:31 smithi196 ceph-mon[76349]: from='client.63213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:32.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1553262983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[100622]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:34:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[102844]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:34:33.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:32 smithi196 ceph-mon[76349]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T00:34:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:33 smithi196 ceph-mon[76349]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:34:34.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:33 smithi184 ceph-mon[100622]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:34:34.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:33 smithi184 ceph-mon[102844]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:34:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:36 smithi184 ceph-mon[100622]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:34:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:36 smithi184 ceph-mon[102844]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:34:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:36 smithi196 ceph-mon[76349]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:34:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:38 smithi184 ceph-mon[100622]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:34:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:38 smithi184 ceph-mon[102844]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:34:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:38 smithi196 ceph-mon[76349]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:34:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:40 smithi184 ceph-mon[100622]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:34:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:40 smithi184 ceph-mon[102844]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:34:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:40 smithi196 ceph-mon[76349]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:34:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:41.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:41.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:41.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:42 smithi184 ceph-mon[100622]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:34:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:42 smithi184 ceph-mon[102844]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:34:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:34:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:42 smithi196 ceph-mon[76349]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:34:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:44 smithi184 ceph-mon[100622]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:34:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:44 smithi184 ceph-mon[102844]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:34:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:44 smithi196 ceph-mon[76349]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:34:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:46 smithi184 ceph-mon[100622]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:34:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:46 smithi184 ceph-mon[102844]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:34:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:46 smithi196 ceph-mon[76349]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:34:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:48 smithi184 ceph-mon[100622]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:48 smithi184 ceph-mon[102844]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:48 smithi196 ceph-mon[76349]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:50 smithi184 ceph-mon[100622]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:50 smithi184 ceph-mon[102844]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:50 smithi196 ceph-mon[76349]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:51.360Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:51.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:51.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:34:51.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:34:51.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:34:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:51.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:34:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:34:52.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[100622]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:52.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[102844]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:52.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:52 smithi196 ceph-mon[76349]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:34:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:54 smithi196 ceph-mon[76349]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:54.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:54 smithi184 ceph-mon[100622]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:54.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:54 smithi184 ceph-mon[102844]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:56 smithi196 ceph-mon[76349]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:56 smithi184 ceph-mon[100622]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:56 smithi184 ceph-mon[102844]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:58 smithi196 ceph-mon[76349]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:58 smithi184 ceph-mon[100622]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:58 smithi184 ceph-mon[102844]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:00 smithi196 ceph-mon[76349]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:00 smithi184 ceph-mon[100622]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:00 smithi184 ceph-mon[102844]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:01.432 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:35:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:01.361Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:01.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:01.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:01.962 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:35:01.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 4h - - 2024-09-21T00:35:01.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 4h - - 2024-09-21T00:35:01.962 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 9s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 9s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 4h 472M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:35:01.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 9s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 4h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 11s ago 4h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 9s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:35:01.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 4h 98.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 9s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 9s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 4h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 9s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:35:01.965 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:35:01.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:35:01.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:35:01.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 9s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:35:02.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:02.272 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:35:02.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:35:02.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:35:02.273 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:35:02.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:35:02.553 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:35:02.554 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:35:02.554 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:35:02.554 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:35:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[100622]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[100622]: from='client.73190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[100622]: from='client.73196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1436008078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[102844]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[102844]: from='client.73190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[102844]: from='client.73196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1436008078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:02.882 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:35:02.882 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:02.882 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:02.882 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:02.883 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:02 smithi196 ceph-mon[76349]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:02 smithi196 ceph-mon[76349]: from='client.73190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:02 smithi196 ceph-mon[76349]: from='client.73196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1436008078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[100622]: from='client.73202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[100622]: from='client.63249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1253273089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[102844]: from='client.73202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[102844]: from='client.63249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1253273089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:03.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:03 smithi196 ceph-mon[76349]: from='client.73202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:03 smithi196 ceph-mon[76349]: from='client.63249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:03.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1253273089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:04 smithi196 ceph-mon[76349]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:04 smithi184 ceph-mon[100622]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:04 smithi184 ceph-mon[102844]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:06 smithi196 ceph-mon[76349]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:06 smithi184 ceph-mon[100622]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:06 smithi184 ceph-mon[102844]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:08 smithi196 ceph-mon[76349]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:08 smithi184 ceph-mon[100622]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:08 smithi184 ceph-mon[102844]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:10 smithi196 ceph-mon[76349]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:10 smithi184 ceph-mon[102844]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:10 smithi184 ceph-mon[100622]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:11.690 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:11.362Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:11.690 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:11.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:11.690 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:11.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[100622]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[102844]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:12.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:12 smithi196 ceph-mon[76349]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:14.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:14 smithi196 ceph-mon[76349]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:14.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:14 smithi184 ceph-mon[100622]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:14 smithi184 ceph-mon[102844]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:16 smithi184 ceph-mon[100622]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:16 smithi184 ceph-mon[102844]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:16 smithi196 ceph-mon[76349]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:18 smithi184 ceph-mon[100622]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:18 smithi184 ceph-mon[102844]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:18 smithi196 ceph-mon[76349]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:20 smithi184 ceph-mon[100622]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:20 smithi184 ceph-mon[102844]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:20 smithi196 ceph-mon[76349]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:21.363Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:21.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:21.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:35:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:22 smithi184 ceph-mon[100622]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:22 smithi184 ceph-mon[102844]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:22 smithi196 ceph-mon[76349]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:24 smithi184 ceph-mon[100622]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:24 smithi184 ceph-mon[102844]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:24 smithi196 ceph-mon[76349]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:26 smithi184 ceph-mon[100622]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:26 smithi184 ceph-mon[102844]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:26 smithi196 ceph-mon[76349]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:28 smithi184 ceph-mon[100622]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:28 smithi184 ceph-mon[102844]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:28 smithi196 ceph-mon[76349]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:30 smithi184 ceph-mon[100622]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:30 smithi184 ceph-mon[102844]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:30 smithi196 ceph-mon[76349]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:31.363Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:31.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:31.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:35:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:32 smithi196 ceph-mon[76349]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[100622]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[102844]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:33.178 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:35:33.738 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:35:33.738 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 4h - - 2024-09-21T00:35:33.738 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 4h - - 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 1s ago 4h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 1s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 4h 473M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 1s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 4h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 1s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:35:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 4h 98.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 1s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 1s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 1s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 1s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 1s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:35:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 1s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:35:34.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:35:34.072 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:35:34.072 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:35:34.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:34.072 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:35:34.073 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:35:34.351 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:35:34.352 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:35:34.686 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:35:34.686 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:34.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:34.686 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:34.686 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: from='client.73226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: from='client.63267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: from='client.63273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/286538670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1624816790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: from='client.73226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: from='client.63267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: from='client.63273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/286538670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:35.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1624816790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: from='client.73226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: from='client.63267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: from='client.63273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/286538670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1624816790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:35 smithi196 ceph-mon[76349]: from='client.73250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:36.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:35 smithi184 ceph-mon[100622]: from='client.73250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:36.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:35 smithi184 ceph-mon[102844]: from='client.73250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:36 smithi184 ceph-mon[100622]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:36 smithi184 ceph-mon[102844]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:36 smithi196 ceph-mon[76349]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:38 smithi196 ceph-mon[76349]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:38 smithi184 ceph-mon[100622]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:38 smithi184 ceph-mon[102844]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:40 smithi196 ceph-mon[76349]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:40 smithi184 ceph-mon[100622]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:40 smithi184 ceph-mon[102844]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:41.364Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:41.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:41.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:35:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:42 smithi196 ceph-mon[76349]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:42 smithi184 ceph-mon[102844]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:42 smithi184 ceph-mon[100622]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:44 smithi184 ceph-mon[100622]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:44 smithi184 ceph-mon[102844]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:44 smithi196 ceph-mon[76349]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:46 smithi196 ceph-mon[76349]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:46 smithi184 ceph-mon[102844]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:46 smithi184 ceph-mon[100622]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:48 smithi196 ceph-mon[76349]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:48 smithi184 ceph-mon[100622]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:48 smithi184 ceph-mon[102844]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:50 smithi196 ceph-mon[76349]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:50 smithi184 ceph-mon[100622]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:50 smithi184 ceph-mon[102844]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:51.365Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:51.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:35:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:35:51.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:35:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:35:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:35:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[100622]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[102844]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:52 smithi196 ceph-mon[76349]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:35:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:54 smithi196 ceph-mon[76349]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:55.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:54 smithi184 ceph-mon[100622]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:55.279 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:54 smithi184 ceph-mon[102844]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:57.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:56 smithi184 ceph-mon[100622]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:57.028 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:56 smithi184 ceph-mon[102844]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:57.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:56 smithi196 ceph-mon[76349]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:57 smithi196 ceph-mon[76349]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:57 smithi184 ceph-mon[100622]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:57 smithi184 ceph-mon[102844]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:00 smithi184 ceph-mon[102844]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:00 smithi184 ceph-mon[100622]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:00 smithi196 ceph-mon[76349]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:01.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:01.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:01.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:02 smithi184 ceph-mon[100622]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:02 smithi184 ceph-mon[102844]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:36:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:02 smithi196 ceph-mon[76349]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:04 smithi184 ceph-mon[100622]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:04 smithi184 ceph-mon[102844]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:04 smithi196 ceph-mon[76349]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:04.985 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:36:05.510 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 4h - - 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 4h - - 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 14s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 4h 473M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 4h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 4h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 14s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:36:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 4h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 4h 98.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:36:05.512 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:36:05.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:36:05.822 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:36:06.089 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:36:06.089 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:36:06.089 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:36:06.090 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:36:06.401 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:36:06.401 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:06.401 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:06.401 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:06.401 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[102844]: from='client.63297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[102844]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[102844]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[102844]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1047703264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[100622]: from='client.63297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[100622]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[100622]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[100622]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1047703264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:06 smithi196 ceph-mon[76349]: from='client.63297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:06 smithi196 ceph-mon[76349]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:06.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:06 smithi196 ceph-mon[76349]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:06 smithi196 ceph-mon[76349]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:06.703 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1047703264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:07 smithi184 ceph-mon[100622]: from='client.73286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1408853776' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:07 smithi184 ceph-mon[102844]: from='client.73286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1408853776' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:07 smithi196 ceph-mon[76349]: from='client.73286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1408853776' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:08 smithi184 ceph-mon[100622]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:08 smithi184 ceph-mon[102844]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:08 smithi196 ceph-mon[76349]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:10 smithi184 ceph-mon[102844]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:10 smithi184 ceph-mon[100622]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:10 smithi196 ceph-mon[76349]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:11.367Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:11.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:11.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:36:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[102844]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:12 smithi184 ceph-mon[100622]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:12 smithi196 ceph-mon[76349]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:15.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:14 smithi184 ceph-mon[100622]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:15.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:15.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:14 smithi184 ceph-mon[102844]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:15.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:14 smithi196 ceph-mon[76349]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:16.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:16 smithi196 ceph-mon[76349]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:16 smithi184 ceph-mon[100622]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:16 smithi184 ceph-mon[102844]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:18 smithi184 ceph-mon[100622]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:18 smithi184 ceph-mon[102844]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:18 smithi196 ceph-mon[76349]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:20 smithi184 ceph-mon[100622]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:20 smithi184 ceph-mon[102844]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:20 smithi196 ceph-mon[76349]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:21.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:21.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:21.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T00:36:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:22 smithi184 ceph-mon[100622]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:22 smithi184 ceph-mon[102844]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:22 smithi196 ceph-mon[76349]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:24 smithi184 ceph-mon[100622]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:24 smithi184 ceph-mon[102844]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:24 smithi196 ceph-mon[76349]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:26 smithi184 ceph-mon[100622]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:26 smithi184 ceph-mon[102844]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:26 smithi196 ceph-mon[76349]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:28 smithi184 ceph-mon[100622]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:28 smithi184 ceph-mon[102844]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:28 smithi196 ceph-mon[76349]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:30 smithi184 ceph-mon[100622]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:30 smithi184 ceph-mon[102844]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:30 smithi196 ceph-mon[76349]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:31.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:31.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:31.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:31.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:31.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[102844]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[100622]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:32 smithi196 ceph-mon[76349]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:34 smithi196 ceph-mon[76349]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:34 smithi184 ceph-mon[100622]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:34 smithi184 ceph-mon[102844]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:36.712 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:36:37.025 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:36 smithi184 ceph-mon[100622]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:37.026 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:36 smithi184 ceph-mon[102844]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:37.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:36 smithi196 ceph-mon[76349]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 4h - - 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 4s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:36:37.259 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 4h 473M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 4h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 4h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 4s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 4h 98.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:36:37.260 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:36:37.261 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:36:37.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:36:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:36:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:36:37.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:36:37.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:36:37.861 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:36:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:37 smithi196 ceph-mon[76349]: from='client.63333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2932522171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:38.189 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:36:38.189 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:38.189 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:38.189 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:38.189 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:38.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:37 smithi184 ceph-mon[100622]: from='client.63333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:38.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2932522171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:38.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:37 smithi184 ceph-mon[102844]: from='client.63333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:38.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2932522171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:38 smithi196 ceph-mon[76349]: from='client.73304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:38 smithi196 ceph-mon[76349]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:38 smithi196 ceph-mon[76349]: from='client.63345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:38 smithi196 ceph-mon[76349]: from='client.63357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2187754530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[100622]: from='client.73304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[100622]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[100622]: from='client.63345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[100622]: from='client.63357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2187754530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[102844]: from='client.73304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[102844]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[102844]: from='client.63345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[102844]: from='client.63357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2187754530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:40 smithi196 ceph-mon[76349]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:40 smithi184 ceph-mon[100622]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:40 smithi184 ceph-mon[102844]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:41.369Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:41.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:41.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:36:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:42 smithi196 ceph-mon[76349]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:42 smithi184 ceph-mon[102844]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:42 smithi184 ceph-mon[100622]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:44 smithi196 ceph-mon[76349]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:44 smithi184 ceph-mon[100622]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:44 smithi184 ceph-mon[102844]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:46 smithi196 ceph-mon[76349]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:46 smithi184 ceph-mon[100622]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:46 smithi184 ceph-mon[102844]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:48 smithi196 ceph-mon[76349]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:48 smithi184 ceph-mon[100622]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:48 smithi184 ceph-mon[102844]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:50 smithi196 ceph-mon[76349]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:50 smithi184 ceph-mon[100622]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:50 smithi184 ceph-mon[102844]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:51.370Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:51.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:36:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:36:51.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:36:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:36:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:36:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:52 smithi196 ceph-mon[76349]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[100622]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[102844]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:36:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:54 smithi184 ceph-mon[100622]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:54 smithi184 ceph-mon[102844]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:54 smithi196 ceph-mon[76349]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:57.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:56 smithi184 ceph-mon[100622]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:57.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:56 smithi184 ceph-mon[102844]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:56 smithi196 ceph-mon[76349]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:59.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:58 smithi196 ceph-mon[76349]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:58 smithi184 ceph-mon[100622]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:58 smithi184 ceph-mon[102844]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:00 smithi184 ceph-mon[100622]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:00 smithi184 ceph-mon[102844]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:00 smithi196 ceph-mon[76349]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:01.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:01.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:01.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:02.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:01 smithi196 ceph-mon[76349]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:01 smithi184 ceph-mon[100622]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:01 smithi184 ceph-mon[102844]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:37:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:04 smithi184 ceph-mon[100622]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:04 smithi184 ceph-mon[102844]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:04 smithi196 ceph-mon[76349]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:06 smithi184 ceph-mon[100622]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:06 smithi184 ceph-mon[102844]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:06 smithi196 ceph-mon[76349]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:37:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:08 smithi184 ceph-mon[100622]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:08 smithi184 ceph-mon[102844]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:08 smithi196 ceph-mon[76349]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:09.019 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:37:09.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 4h - - 2024-09-21T00:37:09.019 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-21T00:37:09.019 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 17s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 93.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 17s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 17s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 17s ago 4h 474M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 4h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 17s ago 4h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 17s ago 4h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 17s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 17s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 17s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 17s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:37:09.020 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:37:09.021 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:37:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:09 smithi184 ceph-mon[100622]: from='client.63369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:09 smithi184 ceph-mon[100622]: from='client.73340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:09 smithi184 ceph-mon[102844]: from='client.63369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:09 smithi184 ceph-mon[102844]: from='client.73340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:37:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:37:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:37:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:37:09.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:37:09.599 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:37:09.600 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:37:09.601 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:37:09.601 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:37:09.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:37:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:09 smithi196 ceph-mon[76349]: from='client.63369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:09 smithi196 ceph-mon[76349]: from='client.73340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:09.912 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:37:09.912 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:09.912 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:09.912 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:09.913 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:10.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4199194951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1798599742' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4199194951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1798599742' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4199194951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1798599742' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:11.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:11.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:11.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:37:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[100622]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[102844]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:12 smithi196 ceph-mon[76349]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:14 smithi196 ceph-mon[76349]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[100622]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[102844]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:16 smithi184 ceph-mon[100622]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:16 smithi184 ceph-mon[102844]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:16 smithi196 ceph-mon[76349]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:18.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:18 smithi196 ceph-mon[76349]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:18.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:18 smithi184 ceph-mon[100622]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:18.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:18 smithi184 ceph-mon[102844]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:20 smithi184 ceph-mon[100622]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:20 smithi184 ceph-mon[102844]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:20 smithi196 ceph-mon[76349]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:21.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:21.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:21.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:22.266 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:37:22.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:22 smithi184 ceph-mon[102844]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:22 smithi184 ceph-mon[100622]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:22 smithi196 ceph-mon[76349]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:24 smithi184 ceph-mon[102844]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:24 smithi184 ceph-mon[100622]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:24 smithi196 ceph-mon[76349]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:26 smithi184 ceph-mon[100622]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:26 smithi184 ceph-mon[102844]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:26 smithi196 ceph-mon[76349]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:28 smithi184 ceph-mon[100622]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:28 smithi184 ceph-mon[102844]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:28 smithi196 ceph-mon[76349]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:30.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:30 smithi184 ceph-mon[100622]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:30 smithi184 ceph-mon[102844]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:30 smithi196 ceph-mon[76349]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:31.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:31.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:31.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:37:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:32 smithi196 ceph-mon[76349]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[100622]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[102844]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[100622]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[102844]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:34 smithi196 ceph-mon[76349]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:36 smithi184 ceph-mon[102844]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:36 smithi184 ceph-mon[100622]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:36 smithi196 ceph-mon[76349]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:38 smithi184 ceph-mon[100622]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:37:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:38 smithi184 ceph-mon[102844]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:37:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:38 smithi196 ceph-mon[76349]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:37:40.218 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:37:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:40 smithi184 ceph-mon[102844]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:40 smithi184 ceph-mon[100622]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:40 smithi196 ceph-mon[76349]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:40.743 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:37:40.743 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 4h - - 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 4h - - 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 9s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 7s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 9s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 9s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 4h 475M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 4h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 9s ago 4h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 7s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:37:40.744 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 9s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 9s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 9s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 9s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 7s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:37:40.745 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:37:41.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:37:41.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:37:41.059 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:37:41.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:37:41.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:37:41.059 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:37:41.332 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:37:41.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:37:41.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[102844]: from='client.73370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[102844]: from='client.73376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[102844]: from='client.63417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/900758850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[100622]: from='client.73370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[100622]: from='client.73376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[100622]: from='client.63417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.334 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/900758850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:41 smithi196 ceph-mon[76349]: from='client.73370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:41 smithi196 ceph-mon[76349]: from='client.73376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:41 smithi196 ceph-mon[76349]: from='client.63417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/900758850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:41.641 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:37:41.641 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:41.641 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:41.641 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:41.642 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:41.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:41.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:41.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:41.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:41.642 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:41.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:42.277 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:37:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[100622]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[100622]: from='client.63429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3695600102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[102844]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[102844]: from='client.63429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3695600102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:42 smithi196 ceph-mon[76349]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:42 smithi196 ceph-mon[76349]: from='client.63429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3695600102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:44 smithi184 ceph-mon[100622]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:44 smithi184 ceph-mon[102844]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:44 smithi196 ceph-mon[76349]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:46 smithi184 ceph-mon[100622]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:46 smithi184 ceph-mon[102844]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:46 smithi196 ceph-mon[76349]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:48 smithi184 ceph-mon[102844]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:48 smithi184 ceph-mon[100622]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:48 smithi196 ceph-mon[76349]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:50 smithi184 ceph-mon[100622]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:50 smithi184 ceph-mon[102844]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:50 smithi196 ceph-mon[76349]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:51.736 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:51.373Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:51.736 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:51.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:37:51.736 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:37:51.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:37:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:37:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:37:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[100622]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[102844]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:52 smithi196 ceph-mon[76349]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:54 smithi196 ceph-mon[76349]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[102844]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[100622]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:37:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:56 smithi196 ceph-mon[76349]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:56 smithi184 ceph-mon[100622]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:56 smithi184 ceph-mon[102844]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:58 smithi184 ceph-mon[100622]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:58 smithi184 ceph-mon[102844]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:58 smithi196 ceph-mon[76349]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:00 smithi196 ceph-mon[76349]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:00.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:00 smithi184 ceph-mon[100622]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:00.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:00.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:00 smithi184 ceph-mon[102844]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:00.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:01.374Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:01.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:01.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T00:38:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:02 smithi196 ceph-mon[76349]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:02 smithi184 ceph-mon[100622]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:02 smithi184 ceph-mon[102844]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:04 smithi196 ceph-mon[76349]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:04 smithi184 ceph-mon[100622]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:04 smithi184 ceph-mon[102844]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:06 smithi196 ceph-mon[76349]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:06 smithi184 ceph-mon[100622]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:06 smithi184 ceph-mon[102844]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:08 smithi196 ceph-mon[76349]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:08 smithi184 ceph-mon[100622]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:08 smithi184 ceph-mon[102844]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:10 smithi196 ceph-mon[76349]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:10 smithi184 ceph-mon[102844]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:10 smithi184 ceph-mon[100622]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:11.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:11.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:11.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:11.947 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:38:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T00:38:12.477 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:38:12.477 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 4h - - 2024-09-21T00:38:12.477 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 0s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 4h 477M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 4h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 0s ago 4h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:38:12.478 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:38:12.479 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:38:12.789 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:38:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:38:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:38:12.791 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:38:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: from='client.63441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3695465527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: from='client.63441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3695465527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:38:13.062 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:38:13.063 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:38:13.063 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:38:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: from='client.63441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3695465527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:13.372 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:38:13.372 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:13.372 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:13.372 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:13.372 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='client.63453 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2351778542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:38:13.811903653Z level=info msg="Completed cleanup jobs" duration=11.667493ms 2024-09-21T00:38:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='client.63453 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2351778542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='client.63453 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2351778542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:14 smithi196 ceph-mon[76349]: from='client.63465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:14 smithi196 ceph-mon[76349]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[100622]: from='client.63465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[100622]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[102844]: from='client.63465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[102844]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:16 smithi196 ceph-mon[76349]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:16 smithi184 ceph-mon[100622]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:16 smithi184 ceph-mon[102844]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:18 smithi184 ceph-mon[102844]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:18 smithi184 ceph-mon[100622]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:18 smithi196 ceph-mon[76349]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:20 smithi184 ceph-mon[100622]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:20 smithi184 ceph-mon[102844]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:20 smithi196 ceph-mon[76349]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:21.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:21.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:21.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:38:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:22 smithi196 ceph-mon[76349]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:22 smithi184 ceph-mon[100622]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:22 smithi184 ceph-mon[102844]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:24 smithi196 ceph-mon[76349]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:24 smithi184 ceph-mon[100622]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:24 smithi184 ceph-mon[102844]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:26 smithi196 ceph-mon[76349]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:26 smithi184 ceph-mon[100622]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:26 smithi184 ceph-mon[102844]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:28 smithi196 ceph-mon[76349]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:28 smithi184 ceph-mon[100622]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:28 smithi184 ceph-mon[102844]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:30 smithi184 ceph-mon[102844]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:30 smithi184 ceph-mon[100622]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:30 smithi196 ceph-mon[76349]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:31.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:31.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:31.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:38:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[100622]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[102844]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:32 smithi196 ceph-mon[76349]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:34 smithi196 ceph-mon[76349]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[102844]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[100622]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:36 smithi196 ceph-mon[76349]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:36 smithi184 ceph-mon[100622]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:36 smithi184 ceph-mon[102844]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:38 smithi196 ceph-mon[76349]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:38 smithi184 ceph-mon[100622]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:38 smithi184 ceph-mon[102844]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:40 smithi196 ceph-mon[76349]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:40 smithi184 ceph-mon[100622]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:40 smithi184 ceph-mon[102844]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:41.377Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:41.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:41.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:38:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:42 smithi196 ceph-mon[76349]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:42 smithi184 ceph-mon[100622]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:42 smithi184 ceph-mon[102844]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:43.667 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:38:44.192 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 4h - - 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 12s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 10s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 12s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 12s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 12s ago 4h 477M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 4h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 4h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 12s ago 4h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 10s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:38:44.193 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 12s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 12s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 12s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 12s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:38:44.194 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:38:44.502 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:38:44.503 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:38:44.772 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:38:44.773 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:38:44.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[100622]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:44.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[100622]: from='client.63477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[100622]: from='client.63483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1875105134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[102844]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[102844]: from='client.63477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[102844]: from='client.63483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1875105134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:44 smithi196 ceph-mon[76349]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:44 smithi196 ceph-mon[76349]: from='client.63477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:44 smithi196 ceph-mon[76349]: from='client.63483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1875105134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:45.080 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:38:45.080 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:45.080 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:45.080 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:45.080 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:45 smithi196 ceph-mon[76349]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:45 smithi196 ceph-mon[76349]: from='client.63501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1888392967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[100622]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[100622]: from='client.63501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1888392967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[102844]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[102844]: from='client.63501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1888392967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:46 smithi196 ceph-mon[76349]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:46 smithi184 ceph-mon[100622]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:46 smithi184 ceph-mon[102844]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:48 smithi196 ceph-mon[76349]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:48 smithi184 ceph-mon[100622]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:48 smithi184 ceph-mon[102844]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:50 smithi196 ceph-mon[76349]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:50 smithi184 ceph-mon[102844]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:50 smithi184 ceph-mon[100622]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:51.378Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:51.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:38:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:38:51.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:38:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:38:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:38:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:52 smithi196 ceph-mon[76349]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[102844]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[100622]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[102844]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[100622]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:54 smithi196 ceph-mon[76349]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:38:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:56 smithi184 ceph-mon[100622]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:56 smithi184 ceph-mon[102844]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:56 smithi196 ceph-mon[76349]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:58 smithi184 ceph-mon[100622]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:58 smithi184 ceph-mon[102844]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:58 smithi196 ceph-mon[76349]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:00.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:00 smithi196 ceph-mon[76349]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:00.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:00 smithi184 ceph-mon[102844]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:00 smithi184 ceph-mon[100622]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:01.476 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:01.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:01.477 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:01.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:01.477 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:01.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:39:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:02 smithi184 ceph-mon[100622]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:39:03.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:02 smithi196 ceph-mon[76349]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:39:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:02 smithi184 ceph-mon[102844]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:39:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:04 smithi196 ceph-mon[76349]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:04 smithi184 ceph-mon[100622]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:04 smithi184 ceph-mon[102844]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:06 smithi184 ceph-mon[100622]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:06 smithi184 ceph-mon[102844]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:06 smithi196 ceph-mon[76349]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:08 smithi184 ceph-mon[100622]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:08 smithi184 ceph-mon[102844]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:08 smithi196 ceph-mon[76349]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:10 smithi184 ceph-mon[100622]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:10 smithi184 ceph-mon[102844]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:10 smithi196 ceph-mon[76349]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:11.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:11.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:11.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:11.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:11.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:11.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:12.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:39:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[100622]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[102844]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:12 smithi196 ceph-mon[76349]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:14 smithi196 ceph-mon[76349]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[100622]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[102844]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:15.376 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 4h - - 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 3s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:39:15.903 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 4h 478M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 4h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 3s ago 4h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:39:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:39:15.905 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:39:15.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:39:15.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:39:15.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:39:15.905 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:39:16.214 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:39:16.214 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:39:16.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:39:16.215 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:39:16.484 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:39:16.484 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:39:16.484 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:39:16.484 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:39:16.485 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:39:16.793 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:39:16.794 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:16.794 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:16.794 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:16.794 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: from='client.73478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: from='client.63519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: from='client.73490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3026021665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/182724600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: from='client.73478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: from='client.63519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: from='client.73490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3026021665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/182724600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: from='client.73478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: from='client.63519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: from='client.73490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3026021665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/182724600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:17 smithi196 ceph-mon[76349]: from='client.63537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:17 smithi184 ceph-mon[102844]: from='client.63537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:17 smithi184 ceph-mon[100622]: from='client.63537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:19.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:18 smithi196 ceph-mon[76349]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:18 smithi184 ceph-mon[100622]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:18 smithi184 ceph-mon[102844]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:21.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:20 smithi184 ceph-mon[100622]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:21.078 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:20 smithi184 ceph-mon[102844]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:20 smithi196 ceph-mon[76349]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:21.380Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:21.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:21.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:22.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:22.132 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:22.132 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:39:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:22 smithi196 ceph-mon[76349]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:23.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:22 smithi184 ceph-mon[100622]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:23.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:22 smithi184 ceph-mon[102844]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:24 smithi196 ceph-mon[76349]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:24 smithi184 ceph-mon[100622]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:24 smithi184 ceph-mon[102844]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:26 smithi196 ceph-mon[76349]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:26 smithi184 ceph-mon[100622]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:26 smithi184 ceph-mon[102844]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:28 smithi196 ceph-mon[76349]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:28 smithi184 ceph-mon[100622]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:28 smithi184 ceph-mon[102844]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:30 smithi196 ceph-mon[76349]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:30 smithi184 ceph-mon[100622]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:31.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:30 smithi184 ceph-mon[102844]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:31.381Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:31.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:31.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T00:39:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:32 smithi196 ceph-mon[76349]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[102844]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[100622]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[100622]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[102844]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:34 smithi196 ceph-mon[76349]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:36 smithi184 ceph-mon[100622]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:36 smithi184 ceph-mon[102844]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:36 smithi196 ceph-mon[76349]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:38 smithi184 ceph-mon[100622]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:38 smithi184 ceph-mon[102844]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:38 smithi196 ceph-mon[76349]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:40 smithi184 ceph-mon[100622]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:40 smithi184 ceph-mon[102844]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:40 smithi196 ceph-mon[76349]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:41.651 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:41.382Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:41.651 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:41.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:41.651 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:41.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:42.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T00:39:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:42 smithi196 ceph-mon[76349]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:43.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:42 smithi184 ceph-mon[100622]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:42 smithi184 ceph-mon[102844]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:44 smithi184 ceph-mon[100622]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:44 smithi184 ceph-mon[102844]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:44 smithi196 ceph-mon[76349]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:46 smithi184 ceph-mon[100622]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:46 smithi184 ceph-mon[102844]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:46 smithi196 ceph-mon[76349]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:47.092 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 4h - - 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 4h - - 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 13s ago 4h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 13s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:39:47.620 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 15s ago 4h 479M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 13s ago 4h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 15s ago 4h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 13s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 15s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 15s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 15s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:39:47.621 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:39:47.622 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 13s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:39:47.622 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:39:47.622 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:39:47.622 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:39:47.622 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:39:47.930 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:39:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:39:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:39:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:39:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:39:47.932 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:39:48.201 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:39:48.202 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:39:48.202 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:39:48.202 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:39:48.513 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:39:48.513 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:48.513 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:48.513 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:48.513 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:48.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[100622]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[100622]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:48.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[100622]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[100622]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3007796017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[102844]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[102844]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[102844]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[102844]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.514 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3007796017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:48 smithi196 ceph-mon[76349]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:48 smithi196 ceph-mon[76349]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:48 smithi196 ceph-mon[76349]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:48 smithi196 ceph-mon[76349]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3007796017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:49 smithi196 ceph-mon[76349]: from='client.63573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3392527214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:49 smithi184 ceph-mon[100622]: from='client.63573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3392527214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:49 smithi184 ceph-mon[102844]: from='client.63573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3392527214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:50 smithi196 ceph-mon[76349]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:50 smithi184 ceph-mon[100622]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:50 smithi184 ceph-mon[102844]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:51.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:51.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:39:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:39:51.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:39:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:39:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:39:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:52 smithi196 ceph-mon[76349]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:52 smithi184 ceph-mon[100622]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:52 smithi184 ceph-mon[102844]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[100622]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[102844]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:54 smithi196 ceph-mon[76349]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:39:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:56 smithi184 ceph-mon[100622]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:56 smithi184 ceph-mon[102844]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:56 smithi196 ceph-mon[76349]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:58 smithi184 ceph-mon[100622]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:58 smithi184 ceph-mon[102844]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:58 smithi196 ceph-mon[76349]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:39:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:00 smithi184 ceph-mon[100622]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:00 smithi184 ceph-mon[102844]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:00 smithi196 ceph-mon[76349]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:01.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:01.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:01.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:40:02.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:02 smithi184 ceph-mon[100622]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:02.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:02 smithi184 ceph-mon[102844]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:02 smithi196 ceph-mon[76349]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:04 smithi196 ceph-mon[76349]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:04.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:04 smithi184 ceph-mon[100622]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:04.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:04 smithi184 ceph-mon[102844]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:06 smithi184 ceph-mon[100622]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:06 smithi184 ceph-mon[102844]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:06 smithi196 ceph-mon[76349]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:08 smithi184 ceph-mon[100622]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:08 smithi184 ceph-mon[102844]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:08 smithi196 ceph-mon[76349]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:10 smithi184 ceph-mon[100622]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:10 smithi184 ceph-mon[102844]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:10 smithi196 ceph-mon[76349]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:11.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:11.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:11.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:40:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:12 smithi184 ceph-mon[102844]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:12 smithi184 ceph-mon[100622]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:12 smithi196 ceph-mon[76349]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:14 smithi196 ceph-mon[76349]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[100622]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[102844]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:16 smithi196 ceph-mon[76349]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:16 smithi184 ceph-mon[100622]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:16 smithi184 ceph-mon[102844]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:18.811 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:40:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:18 smithi184 ceph-mon[102844]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:18 smithi184 ceph-mon[100622]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:18 smithi196 ceph-mon[76349]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 4h - - 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 4h - - 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 6s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 5s ago 4h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:40:19.337 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 4h 480M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 5s ago 4h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 4h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 6s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 5s ago 4h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 5s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 5s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 5s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:40:19.338 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 5s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:40:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 5s ago 4h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:40:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:40:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:40:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:40:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 5s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:40:19.645 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:40:19.646 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:40:19.919 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:19 smithi184 ceph-mon[100622]: from='client.73550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:19.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:19 smithi184 ceph-mon[102844]: from='client.73550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:19.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:40:19.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:40:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:19 smithi196 ceph-mon[76349]: from='client.73550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:20.229 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:20.229 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:20.229 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:20.229 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:20.229 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3852686229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: from='client.63609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/765040974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3852686229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: from='client.63609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/765040974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3852686229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: from='client.63609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/765040974' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:21.657 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:21.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:21.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:21.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:21.658 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:21.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:22.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:40:22.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:22 smithi184 ceph-mon[100622]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:22.983 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:22 smithi184 ceph-mon[102844]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:22 smithi196 ceph-mon[76349]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:24.702 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:24 smithi196 ceph-mon[76349]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:24 smithi184 ceph-mon[100622]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:24 smithi184 ceph-mon[102844]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:26 smithi184 ceph-mon[100622]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:26 smithi184 ceph-mon[102844]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:26 smithi196 ceph-mon[76349]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:28 smithi184 ceph-mon[100622]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:28 smithi184 ceph-mon[102844]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:28 smithi196 ceph-mon[76349]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:30 smithi184 ceph-mon[100622]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:30 smithi184 ceph-mon[102844]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:30 smithi196 ceph-mon[76349]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:31.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:31.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:31.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:31.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:31.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:31.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:40:33.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:32 smithi196 ceph-mon[76349]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:32 smithi184 ceph-mon[100622]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:32 smithi184 ceph-mon[102844]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[100622]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[102844]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:34 smithi196 ceph-mon[76349]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:36 smithi184 ceph-mon[100622]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:36 smithi184 ceph-mon[102844]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:36 smithi196 ceph-mon[76349]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:38 smithi184 ceph-mon[100622]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:38 smithi184 ceph-mon[102844]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:38 smithi196 ceph-mon[76349]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:40 smithi184 ceph-mon[100622]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:40 smithi184 ceph-mon[102844]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:40 smithi196 ceph-mon[76349]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:41.780 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:41.387Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:41.781 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:41.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:41.781 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:41.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:40:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:42 smithi184 ceph-mon[102844]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:42 smithi184 ceph-mon[100622]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:42 smithi196 ceph-mon[76349]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:45.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:44 smithi184 ceph-mon[102844]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:45.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:45.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:44 smithi184 ceph-mon[100622]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:45.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:45.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:44 smithi196 ceph-mon[76349]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:45.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:46 smithi184 ceph-mon[100622]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:46 smithi184 ceph-mon[102844]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:47.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:46 smithi196 ceph-mon[76349]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:40:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:48 smithi184 ceph-mon[100622]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:48 smithi184 ceph-mon[102844]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:48 smithi196 ceph-mon[76349]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:40:50.524 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:40:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:50 smithi184 ceph-mon[100622]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:50 smithi184 ceph-mon[102844]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 4h - - 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 4h - - 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 18s ago 4h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 16s ago 4h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:40:51.051 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 18s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 4h 481M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 4h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 4h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 18s ago 4h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 16s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:40:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 4h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:40:51.053 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:40:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:50 smithi196 ceph-mon[76349]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:40:51.363 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:40:51.631 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:40:51.631 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:40:51.632 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:40:51.633 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:40:51.633 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:40:51.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:51.387Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:51.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:51.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:40:51.633 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:40:51.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:40:51.944 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:51.944 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:51.944 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:51.944 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:51.944 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:51.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:51.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[100622]: from='client.73586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:51.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[100622]: from='client.73592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:51.944 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1864795102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:51.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:51.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[102844]: from='client.73586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:51.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[102844]: from='client.73592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:51.945 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1864795102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:51 smithi196 ceph-mon[76349]: from='client.73586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:51 smithi196 ceph-mon[76349]: from='client.73592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1864795102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:40:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:40:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:52 smithi196 ceph-mon[76349]: from='client.73598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:52 smithi196 ceph-mon[76349]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:52 smithi196 ceph-mon[76349]: from='client.73610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1072146197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[100622]: from='client.73598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[100622]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[100622]: from='client.73610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1072146197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[102844]: from='client.73598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[102844]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[102844]: from='client.73610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1072146197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:54 smithi196 ceph-mon[76349]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:40:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[100622]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[102844]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:40:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:56 smithi184 ceph-mon[100622]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:56 smithi184 ceph-mon[102844]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:57.333 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:56 smithi196 ceph-mon[76349]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:40:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:58 smithi184 ceph-mon[100622]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:58 smithi184 ceph-mon[102844]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:40:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:58 smithi196 ceph-mon[76349]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:41:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:00 smithi184 ceph-mon[100622]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:41:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:00 smithi184 ceph-mon[102844]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:41:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:00 smithi196 ceph-mon[76349]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:41:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:01.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:01.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:01.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:41:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:03.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:02 smithi196 ceph-mon[76349]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:02 smithi184 ceph-mon[100622]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:02 smithi184 ceph-mon[102844]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:05.174 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:04 smithi184 ceph-mon[100622]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:04 smithi184 ceph-mon[102844]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:05.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:04 smithi196 ceph-mon[76349]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:41:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:06 smithi184 ceph-mon[100622]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:06 smithi184 ceph-mon[102844]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:06 smithi196 ceph-mon[76349]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:08 smithi184 ceph-mon[102844]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:41:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:08 smithi184 ceph-mon[100622]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:41:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:08 smithi196 ceph-mon[76349]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:41:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:10 smithi184 ceph-mon[100622]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:10 smithi184 ceph-mon[102844]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:10 smithi196 ceph-mon[76349]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:11.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:11.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:11.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:41:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:12 smithi184 ceph-mon[100622]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:12 smithi184 ceph-mon[102844]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:12 smithi196 ceph-mon[76349]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[100622]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[102844]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:15 smithi196 ceph-mon[76349]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:16 smithi184 ceph-mon[100622]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:16 smithi184 ceph-mon[102844]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:16 smithi196 ceph-mon[76349]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:18 smithi196 ceph-mon[76349]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:18 smithi184 ceph-mon[100622]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:18 smithi184 ceph-mon[102844]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:20 smithi196 ceph-mon[76349]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:20 smithi184 ceph-mon[100622]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:20 smithi184 ceph-mon[102844]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:21.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:21.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:21.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:22.244 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:41:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:41:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:22 smithi196 ceph-mon[76349]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:22.769 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:41:22.769 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 4h - - 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 4h - - 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 9s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 9s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 9s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 4h 480M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 8s ago 4h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 4h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 9s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:41:22.770 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 9s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 9s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 9s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 9s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 8s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 8s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 8s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 8s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 74.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:41:22.771 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 8s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:41:22.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:22 smithi184 ceph-mon[100622]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:22.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:22 smithi184 ceph-mon[102844]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:23.077 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:41:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:41:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:41:23.078 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:41:23.347 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:41:23.348 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:41:23.348 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:41:23.348 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:41:23.348 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:41:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:23 smithi196 ceph-mon[76349]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:23 smithi196 ceph-mon[76349]: from='client.73628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:23 smithi196 ceph-mon[76349]: from='client.73634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1721920896' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:23.658 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:41:23.658 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:23.658 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:23.658 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:23.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[100622]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.658 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[100622]: from='client.73628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[100622]: from='client.73634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1721920896' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[102844]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[102844]: from='client.73628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[102844]: from='client.73634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:23.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1721920896' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:24 smithi196 ceph-mon[76349]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:24 smithi196 ceph-mon[76349]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3078956648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:24.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[100622]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:24.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[100622]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:24.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3078956648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[102844]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[102844]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:24.773 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3078956648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:26.376 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:26 smithi196 ceph-mon[76349]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:26 smithi184 ceph-mon[100622]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:26 smithi184 ceph-mon[102844]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:28 smithi196 ceph-mon[76349]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:28 smithi184 ceph-mon[100622]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:28 smithi184 ceph-mon[102844]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:30 smithi196 ceph-mon[76349]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:30 smithi184 ceph-mon[100622]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:30 smithi184 ceph-mon[102844]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:31.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:31.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:31.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:41:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:32 smithi196 ceph-mon[76349]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:32 smithi184 ceph-mon[100622]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:32 smithi184 ceph-mon[102844]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:34 smithi196 ceph-mon[76349]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[100622]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:34 smithi184 ceph-mon[102844]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:36 smithi184 ceph-mon[100622]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:36 smithi184 ceph-mon[102844]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:36 smithi196 ceph-mon[76349]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:38 smithi184 ceph-mon[102844]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:38 smithi184 ceph-mon[100622]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:38 smithi196 ceph-mon[76349]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:40 smithi184 ceph-mon[100622]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:40 smithi184 ceph-mon[102844]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:40 smithi196 ceph-mon[76349]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:41.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:41.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:41.773 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:41.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:42.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:41:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:42 smithi184 ceph-mon[100622]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:42 smithi184 ceph-mon[102844]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:42 smithi196 ceph-mon[76349]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:44 smithi184 ceph-mon[100622]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:44 smithi184 ceph-mon[102844]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:44 smithi196 ceph-mon[76349]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:46.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:46 smithi184 ceph-mon[100622]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:46.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:46 smithi184 ceph-mon[102844]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:47.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:46 smithi196 ceph-mon[76349]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:48 smithi184 ceph-mon[100622]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:48 smithi184 ceph-mon[102844]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:48 smithi196 ceph-mon[76349]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:50 smithi184 ceph-mon[100622]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:50 smithi184 ceph-mon[102844]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:50 smithi196 ceph-mon[76349]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:51.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:51.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:51.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:51.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:41:51.794 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:41:51.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:41:52.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:52.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:52.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:41:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:41:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:52 smithi184 ceph-mon[100622]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:52 smithi184 ceph-mon[102844]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:52 smithi196 ceph-mon[76349]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:53.947 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 4h - - 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 4h - - 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 1s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 19s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 1s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 19s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:41:54.475 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 1s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 1s ago 4h 481M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 4h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 1s ago 4h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 1s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 19s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 1s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 1s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 1s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 1s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 19s ago 4h 74.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:41:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:41:54.477 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:41:54.477 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:41:54.477 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[100622]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[100622]: from='client.63693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[102844]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:54 smithi184 ceph-mon[102844]: from='client.63693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:54 smithi196 ceph-mon[76349]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:54 smithi196 ceph-mon[76349]: from='client.63693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:54.784 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:41:54.784 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:41:54.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:41:54.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:41:54.785 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:41:55.056 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:41:55.057 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:41:55.375 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:41:55.375 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:55.375 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:55.375 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:55.375 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:55.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='client.63699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='client.73670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/618069225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='client.63699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='client.73670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/618069225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='client.63699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='client.73670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/618069225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:41:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[100622]: from='client.73682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[100622]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/508842547' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[102844]: from='client.73682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[102844]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/508842547' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:56 smithi196 ceph-mon[76349]: from='client.73682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:56 smithi196 ceph-mon[76349]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/508842547' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:58 smithi196 ceph-mon[76349]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:58 smithi184 ceph-mon[100622]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:58 smithi184 ceph-mon[102844]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:00 smithi184 ceph-mon[100622]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:00 smithi184 ceph-mon[102844]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:00 smithi196 ceph-mon[76349]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:01.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:01.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:01.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T00:42:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:02 smithi196 ceph-mon[76349]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:02.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:02 smithi184 ceph-mon[100622]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:02.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:02 smithi184 ceph-mon[102844]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:04 smithi196 ceph-mon[76349]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:04 smithi184 ceph-mon[100622]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:04 smithi184 ceph-mon[102844]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:06.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:06 smithi196 ceph-mon[76349]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:06.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:06 smithi184 ceph-mon[100622]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:06.690 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:06 smithi184 ceph-mon[102844]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:08 smithi196 ceph-mon[76349]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:08 smithi184 ceph-mon[100622]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:08 smithi184 ceph-mon[102844]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:10 smithi184 ceph-mon[100622]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:10 smithi184 ceph-mon[102844]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:10 smithi196 ceph-mon[76349]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:11.393Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:11.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:11.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T00:42:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:12 smithi196 ceph-mon[76349]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:12 smithi184 ceph-mon[100622]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:12 smithi184 ceph-mon[102844]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[100622]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[102844]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:14 smithi196 ceph-mon[76349]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:16.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:17 smithi184 ceph-mon[100622]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:17 smithi184 ceph-mon[102844]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:17 smithi196 ceph-mon[76349]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:19 smithi196 ceph-mon[76349]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:19 smithi184 ceph-mon[100622]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:19 smithi184 ceph-mon[102844]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:21 smithi184 ceph-mon[100622]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:21 smithi184 ceph-mon[102844]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:21 smithi196 ceph-mon[76349]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:21.394Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:21.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:21.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:42:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:23 smithi184 ceph-mon[100622]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:23 smithi184 ceph-mon[102844]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:23 smithi196 ceph-mon[76349]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:24 smithi184 ceph-mon[100622]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:24 smithi184 ceph-mon[102844]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:24 smithi196 ceph-mon[76349]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:25.669 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 4h - - 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 4h - - 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 12s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 11s ago 4h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 12s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 11s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 12s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:42:26.199 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 12s ago 5h 482M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 11s ago 4h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 4h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 12s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 11s ago 4h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 12s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 12s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 12s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 12s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 11s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 11s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 11s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 11s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 11s ago 4h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:42:26.200 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:42:26.201 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:42:26.201 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 11s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[100622]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[100622]: from='client.63729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[100622]: from='client.73700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[102844]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[102844]: from='client.63729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:26 smithi184 ceph-mon[102844]: from='client.73700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:42:26.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:42:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:26 smithi196 ceph-mon[76349]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:26 smithi196 ceph-mon[76349]: from='client.63729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:26 smithi196 ceph-mon[76349]: from='client.73700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:26.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:42:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:42:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:42:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:42:26.800 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:42:27.114 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:42:27.114 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:27.114 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:27.114 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:27.114 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:27 smithi196 ceph-mon[76349]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1696119565' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:27 smithi196 ceph-mon[76349]: from='client.73718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2715033461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[100622]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1696119565' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[100622]: from='client.73718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2715033461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[102844]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1696119565' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:27.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[102844]: from='client.73718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:27.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2715033461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:28 smithi196 ceph-mon[76349]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:28.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:28 smithi184 ceph-mon[100622]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:28.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:28 smithi184 ceph-mon[102844]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:30 smithi184 ceph-mon[100622]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:30 smithi184 ceph-mon[102844]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:30 smithi196 ceph-mon[76349]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:31.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:31.395Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:31.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:31.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:31.684 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:31.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:42:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:32 smithi184 ceph-mon[100622]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:32 smithi184 ceph-mon[102844]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:32 smithi196 ceph-mon[76349]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[100622]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[102844]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:34 smithi196 ceph-mon[76349]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:36 smithi196 ceph-mon[76349]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[100622]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:36 smithi184 ceph-mon[102844]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:38 smithi196 ceph-mon[76349]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:38 smithi184 ceph-mon[100622]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:38 smithi184 ceph-mon[102844]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:40 smithi184 ceph-mon[100622]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:40 smithi184 ceph-mon[102844]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:40 smithi196 ceph-mon[76349]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:41.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:41.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:41.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:42:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:42 smithi196 ceph-mon[76349]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:42.695 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:42 smithi184 ceph-mon[100622]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:42.695 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:42 smithi184 ceph-mon[102844]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:44 smithi196 ceph-mon[76349]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:44 smithi184 ceph-mon[100622]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:44 smithi184 ceph-mon[102844]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:46 smithi196 ceph-mon[76349]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:46 smithi184 ceph-mon[100622]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:46 smithi184 ceph-mon[102844]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:48.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:48 smithi184 ceph-mon[100622]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:48.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:48 smithi184 ceph-mon[102844]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:48 smithi196 ceph-mon[76349]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:50 smithi184 ceph-mon[100622]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:50 smithi184 ceph-mon[102844]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:50 smithi196 ceph-mon[76349]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:51.397Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:51.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:42:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:42:51.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:42:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:42:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:42:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:52 smithi184 ceph-mon[100622]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:52 smithi184 ceph-mon[102844]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:52 smithi196 ceph-mon[76349]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[100622]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[102844]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:54 smithi196 ceph-mon[76349]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[100622]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:56 smithi184 ceph-mon[102844]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:42:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:56 smithi196 ceph-mon[76349]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:57.411 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:42:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:42:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 4h - - 2024-09-21T00:42:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 4h - - 2024-09-21T00:42:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 4s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:42:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 2s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 4s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 2s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 4s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 4s ago 5h 483M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 2s ago 4h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 4s ago 4h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 4s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 2s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 4s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 4s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 4s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 4s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:42:57.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 2s ago 4h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:42:57.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:42:58.246 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:42:58.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:42:58.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:42:58.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:42:58.518 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:42:58.519 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[100622]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[100622]: from='client.73730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[100622]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[100622]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/753358082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[102844]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[102844]: from='client.73730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.519 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[102844]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[102844]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/753358082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:58 smithi196 ceph-mon[76349]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:58 smithi196 ceph-mon[76349]: from='client.73730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:58 smithi196 ceph-mon[76349]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:58 smithi196 ceph-mon[76349]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/753358082' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:58.826 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:42:58.827 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:58.827 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:58.827 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:58.827 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:59 smithi196 ceph-mon[76349]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/622512345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:59.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[100622]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:59.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/622512345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:59.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:59.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[102844]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:59.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/622512345' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:59.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:00 smithi184 ceph-mon[100622]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:00 smithi184 ceph-mon[102844]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:00 smithi196 ceph-mon[76349]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:01.398Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:01.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:01.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:43:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:02 smithi196 ceph-mon[76349]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:02 smithi184 ceph-mon[100622]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:02 smithi184 ceph-mon[102844]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:04 smithi196 ceph-mon[76349]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:04 smithi184 ceph-mon[100622]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:04 smithi184 ceph-mon[102844]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:06 smithi184 ceph-mon[100622]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:06 smithi184 ceph-mon[102844]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:06 smithi196 ceph-mon[76349]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:08.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:08 smithi184 ceph-mon[102844]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:08.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:08 smithi184 ceph-mon[100622]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:08 smithi196 ceph-mon[76349]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:10.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:10 smithi196 ceph-mon[76349]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:10 smithi184 ceph-mon[100622]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:10 smithi184 ceph-mon[102844]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:11.398Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:11.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:11.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:43:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:12 smithi184 ceph-mon[100622]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:12 smithi184 ceph-mon[102844]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:12 smithi196 ceph-mon[76349]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[100622]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[102844]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:14 smithi196 ceph-mon[76349]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[100622]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[102844]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:16 smithi196 ceph-mon[76349]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:18 smithi184 ceph-mon[100622]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:18 smithi184 ceph-mon[102844]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:18 smithi196 ceph-mon[76349]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:20 smithi184 ceph-mon[100622]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:20 smithi184 ceph-mon[102844]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:20 smithi196 ceph-mon[76349]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:21.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:21.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:21.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:21.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:43:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:22 smithi184 ceph-mon[102844]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:22 smithi184 ceph-mon[100622]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:22 smithi196 ceph-mon[76349]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:24 smithi184 ceph-mon[102844]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:24 smithi184 ceph-mon[100622]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:24 smithi196 ceph-mon[76349]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:26 smithi184 ceph-mon[102844]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:26 smithi184 ceph-mon[100622]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:26 smithi196 ceph-mon[76349]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:28.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:28 smithi184 ceph-mon[100622]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:28.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:28 smithi184 ceph-mon[102844]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:28.781 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:28 smithi196 ceph-mon[76349]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:29.125 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:43:29.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:29.665 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 4h - - 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 15s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 14s ago 4h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 15s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 14s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 15s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:43:29.666 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 15s ago 5h 484M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 14s ago 4h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 15s ago 4h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 15s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 14s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 15s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 15s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 15s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:43:29.667 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 14s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 14s ago 4h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 14s ago 4h 77.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:43:29.668 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 14s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:43:29.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:29.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:29.989 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:43:29.989 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:43:29.989 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:43:29.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:43:29.989 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:43:29.990 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:43:30.286 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:43:30.287 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:43:30.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:30 smithi196 ceph-mon[76349]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:30 smithi196 ceph-mon[76349]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:30.469 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:30 smithi196 ceph-mon[76349]: from='client.63807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:30 smithi196 ceph-mon[76349]: from='client.63813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2499031554' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[102844]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[102844]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[102844]: from='client.63807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[102844]: from='client.63813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2499031554' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[100622]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[100622]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[100622]: from='client.63807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[100622]: from='client.63813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2499031554' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:30.635 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:43:30.636 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:30.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:30.636 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:30.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[102844]: from='client.63825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1035932005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[100622]: from='client.63825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1035932005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:31.400Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:31.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:31.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:31.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:31.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:31 smithi196 ceph-mon[76349]: from='client.63825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1035932005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:43:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:32 smithi184 ceph-mon[100622]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:32 smithi184 ceph-mon[102844]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:32 smithi196 ceph-mon[76349]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[100622]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[102844]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:35.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:34 smithi196 ceph-mon[76349]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:35.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[100622]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[102844]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:36 smithi196 ceph-mon[76349]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:38 smithi184 ceph-mon[100622]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:38 smithi184 ceph-mon[102844]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:38 smithi196 ceph-mon[76349]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:40 smithi184 ceph-mon[100622]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:40 smithi184 ceph-mon[102844]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:40 smithi196 ceph-mon[76349]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:41.401Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 9 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:41.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:41.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:43:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:42 smithi184 ceph-mon[100622]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:42 smithi184 ceph-mon[102844]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:42 smithi196 ceph-mon[76349]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:44 smithi184 ceph-mon[102844]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:44 smithi184 ceph-mon[100622]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:44 smithi196 ceph-mon[76349]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:46 smithi184 ceph-mon[100622]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:46 smithi184 ceph-mon[102844]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:46 smithi196 ceph-mon[76349]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:48 smithi184 ceph-mon[100622]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:48 smithi184 ceph-mon[102844]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:48 smithi196 ceph-mon[76349]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:50.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:50 smithi196 ceph-mon[76349]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:50 smithi184 ceph-mon[100622]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:50 smithi184 ceph-mon[102844]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:51.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:51.649 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:51.649 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:51.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:51.649 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:51.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:43:51.649 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:43:51.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:43:51.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:43:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:43:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:52 smithi196 ceph-mon[76349]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:52 smithi184 ceph-mon[100622]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:52 smithi184 ceph-mon[102844]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:54 smithi196 ceph-mon[76349]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[102844]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[100622]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[100622]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[102844]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:56 smithi196 ceph-mon[76349]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:43:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:58 smithi184 ceph-mon[100622]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:58 smithi184 ceph-mon[102844]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:58 smithi196 ceph-mon[76349]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:00.932 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:44:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:00 smithi184 ceph-mon[100622]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:00 smithi184 ceph-mon[102844]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:00 smithi196 ceph-mon[76349]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:01.463 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:44:01.463 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 7s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 5s ago 4h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 7s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 5s ago 4h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 7s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 7s ago 5h 484M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 5s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 7s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 7s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 5s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 7s ago 4h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:44:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 4h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 5s ago 4h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 5s ago 4h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 5s ago 4h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 5s ago 4h 96.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 5s ago 4h 77.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:44:01.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 5s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:44:01.701 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:01.403Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:01.702 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:01.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:01.702 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:01.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:01.779 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:44:01.780 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:44:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-mon[102844]: from='client.63837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:01 smithi184 ceph-mon[100622]: from='client.63837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:02.054 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:44:02.055 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:44:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:01 smithi196 ceph-mon[76349]: from='client.63837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:44:02.374 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:44:02.375 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:02.375 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:02.375 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:02.375 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[100622]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[100622]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[100622]: from='client.73814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1361013002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2124315243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[102844]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[102844]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[102844]: from='client.73814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1361013002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2124315243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:02 smithi196 ceph-mon[76349]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:02 smithi196 ceph-mon[76349]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:02 smithi196 ceph-mon[76349]: from='client.73814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1361013002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2124315243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:03 smithi184 ceph-mon[100622]: from='client.73826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:03 smithi184 ceph-mon[102844]: from='client.73826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:03 smithi196 ceph-mon[76349]: from='client.73826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:05.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:04 smithi196 ceph-mon[76349]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:04 smithi184 ceph-mon[100622]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:04 smithi184 ceph-mon[102844]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:06 smithi184 ceph-mon[100622]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:44:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:06 smithi184 ceph-mon[102844]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:44:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:06 smithi196 ceph-mon[76349]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:44:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:08 smithi184 ceph-mon[100622]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:44:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:08 smithi184 ceph-mon[102844]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:44:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:08 smithi196 ceph-mon[76349]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:44:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:10 smithi196 ceph-mon[76349]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:10 smithi184 ceph-mon[100622]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:10 smithi184 ceph-mon[102844]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:11.728 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:11.403Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:11.728 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:11.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:11.728 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:11.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:12.087 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:44:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:12 smithi184 ceph-mon[102844]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:12 smithi184 ceph-mon[100622]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:12 smithi196 ceph-mon[76349]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[100622]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[102844]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:14 smithi196 ceph-mon[76349]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[100622]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[102844]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:16 smithi196 ceph-mon[76349]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T00:44:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:18 smithi184 ceph-mon[100622]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:18 smithi184 ceph-mon[102844]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:18 smithi196 ceph-mon[76349]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:20 smithi184 ceph-mon[100622]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:44:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:20 smithi184 ceph-mon[102844]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:44:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:20 smithi196 ceph-mon[76349]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:44:21.788 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:21.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:21.788 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:21.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:21.788 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:21.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:22.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:44:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:22 smithi184 ceph-mon[102844]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:22 smithi184 ceph-mon[100622]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:22 smithi196 ceph-mon[76349]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T00:44:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:24 smithi184 ceph-mon[102844]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:44:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:24 smithi184 ceph-mon[100622]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:44:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:24 smithi196 ceph-mon[76349]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:44:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:26 smithi184 ceph-mon[102844]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:44:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:26 smithi184 ceph-mon[100622]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:44:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:26 smithi196 ceph-mon[76349]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T00:44:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:28 smithi196 ceph-mon[76349]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:28 smithi184 ceph-mon[102844]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:28 smithi184 ceph-mon[100622]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:30.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:30.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:30 smithi184 ceph-mon[102844]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:44:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:30 smithi184 ceph-mon[100622]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:44:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:30 smithi196 ceph-mon[76349]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:44:31.775 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:31.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:31.775 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:31.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:31.775 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:31.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:32.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:44:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:32.715 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:44:32.844 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:32 smithi184 ceph-mon[100622]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:44:32.845 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:32 smithi184 ceph-mon[102844]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:44:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:32 smithi196 ceph-mon[76349]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:44:33.267 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:44:33.267 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T00:44:33.267 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T00:44:33.267 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 17s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 5h 485M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 17s ago 5h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 17s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 4h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 4h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 4h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:44:33.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 4h 96.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 17s ago 4h 78.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:44:33.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:44:33.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:44:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:44:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:44:33.587 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:44:33.857 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:44:33.857 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:44:33.857 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:44:33.857 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:44:33.857 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:44:33.858 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:44:33.858 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[100622]: from='client.73838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:33.858 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[100622]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:33.859 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4113253540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:33.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[102844]: from='client.73838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:33.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[102844]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:33.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4113253540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:33 smithi196 ceph-mon[76349]: from='client.73838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:33 smithi196 ceph-mon[76349]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4113253540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:34.170 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:44:34.171 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:34.171 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2342695968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2342695968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2342695968' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:35.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[102844]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[100622]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:36 smithi196 ceph-mon[76349]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T00:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:38 smithi184 ceph-mon[100622]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:44:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:38 smithi184 ceph-mon[102844]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:44:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:38 smithi196 ceph-mon[76349]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:40 smithi184 ceph-mon[100622]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:40 smithi184 ceph-mon[102844]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:40 smithi196 ceph-mon[76349]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T00:44:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:41.405Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:41.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:41.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:44:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:42 smithi184 ceph-mon[102844]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T00:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:42 smithi184 ceph-mon[100622]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T00:44:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:42 smithi196 ceph-mon[76349]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T00:44:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:44 smithi184 ceph-mon[100622]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:44:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:44 smithi184 ceph-mon[102844]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:44:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:44 smithi196 ceph-mon[76349]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:44:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:46 smithi184 ceph-mon[100622]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:44:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:46 smithi184 ceph-mon[102844]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:44:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:46 smithi196 ceph-mon[76349]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T00:44:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:48 smithi184 ceph-mon[100622]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:48 smithi184 ceph-mon[102844]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:48 smithi196 ceph-mon[76349]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T00:44:51.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:50 smithi196 ceph-mon[76349]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:50 smithi184 ceph-mon[100622]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:50 smithi184 ceph-mon[102844]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:51.722 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:51.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:51.722 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:51.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:44:51.722 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:44:51.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:44:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:44:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:44:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:52 smithi184 ceph-mon[100622]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:44:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:52 smithi184 ceph-mon[102844]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:44:53.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:52 smithi196 ceph-mon[76349]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:44:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[100622]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[102844]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:54 smithi196 ceph-mon[76349]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[100622]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[102844]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:57 smithi196 ceph-mon[76349]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:44:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:58 smithi196 ceph-mon[76349]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:44:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:58 smithi184 ceph-mon[102844]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:44:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:58 smithi184 ceph-mon[100622]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:44:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:44:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:44:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:44:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:00 smithi196 ceph-mon[76349]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:00 smithi184 ceph-mon[102844]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:00 smithi184 ceph-mon[100622]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:01.407 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:01.407 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:01.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:01.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:01.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:02 smithi184 ceph-mon[100622]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:02 smithi184 ceph-mon[102844]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:02] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:45:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:02 smithi196 ceph-mon[76349]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:04.471 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:45:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:04 smithi184 ceph-mon[100622]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:04 smithi184 ceph-mon[102844]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:04 smithi196 ceph-mon[76349]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:04.994 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:45:04.994 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 5h - - 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 5h - - 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 10s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 8s ago 4h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 10s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 10s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 10s ago 5h 485M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 8s ago 5h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 10s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 10s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 8s ago 4h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 10s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 10s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:45:04.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 4h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 8s ago 4h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 8s ago 4h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 8s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 8s ago 4h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 8s ago 4h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:45:04.996 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 8s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:45:05.304 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:05.305 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:45:05.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:45:05.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:45:05.306 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:45:05.576 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:45:05.577 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[100622]: from='client.63909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[100622]: from='client.73880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[100622]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3250454328' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[102844]: from='client.63909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[102844]: from='client.73880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[102844]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.578 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3250454328' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:05.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:05 smithi196 ceph-mon[76349]: from='client.63909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:05 smithi196 ceph-mon[76349]: from='client.73880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:05 smithi196 ceph-mon[76349]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:05.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3250454328' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[100622]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[100622]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2789827028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[102844]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[102844]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2789827028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:06 smithi196 ceph-mon[76349]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:06 smithi196 ceph-mon[76349]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2789827028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:08 smithi184 ceph-mon[102844]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:08 smithi184 ceph-mon[100622]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:08 smithi196 ceph-mon[76349]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:10 smithi184 ceph-mon[100622]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:10 smithi184 ceph-mon[102844]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:10 smithi196 ceph-mon[76349]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:11.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:11.407Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:11.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:11.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:12.288 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:12] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:45:12.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:12 smithi184 ceph-mon[100622]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:12 smithi184 ceph-mon[102844]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:12.818 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:12 smithi196 ceph-mon[76349]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:14 smithi184 ceph-mon[100622]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:14.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:14 smithi184 ceph-mon[102844]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:14 smithi196 ceph-mon[76349]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[102844]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[100622]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:16 smithi196 ceph-mon[76349]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:18 smithi184 ceph-mon[100622]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:18 smithi184 ceph-mon[102844]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:18 smithi196 ceph-mon[76349]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:20 smithi184 ceph-mon[100622]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:20 smithi184 ceph-mon[102844]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:20 smithi196 ceph-mon[76349]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:21.408Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:21.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:21.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:45:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:22 smithi184 ceph-mon[100622]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:22 smithi184 ceph-mon[102844]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:22 smithi196 ceph-mon[76349]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:24 smithi184 ceph-mon[102844]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:24 smithi184 ceph-mon[100622]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:24 smithi196 ceph-mon[76349]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:26 smithi196 ceph-mon[76349]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:26 smithi184 ceph-mon[100622]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:26 smithi184 ceph-mon[102844]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:28 smithi196 ceph-mon[76349]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:28 smithi184 ceph-mon[100622]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:28 smithi184 ceph-mon[102844]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:30 smithi184 ceph-mon[100622]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:30 smithi184 ceph-mon[102844]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:30 smithi196 ceph-mon[76349]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:31.409Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:31.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:31.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:45:32.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:32 smithi196 ceph-mon[76349]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:32.892 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:32 smithi184 ceph-mon[100622]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:32.892 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:32 smithi184 ceph-mon[102844]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:34 smithi196 ceph-mon[76349]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:34 smithi184 ceph-mon[100622]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:34 smithi184 ceph-mon[102844]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:45:36.709 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:45:36.709 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 5h - - 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 2s ago 4h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 0s ago 4h 91.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 0s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 5h 486M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 0s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 2s ago 4h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 0s ago 4h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:45:36.710 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 4h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 0s ago 4h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 0s ago 4h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 0s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 4h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 0s ago 4h 78.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:45:36.711 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 0s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[102844]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[100622]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:45:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:45:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:45:37.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:45:37.020 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:45:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:36 smithi196 ceph-mon[76349]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:37.285 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:45:37.286 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:45:37.593 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:45:37.594 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:37.594 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:37.594 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:37.594 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[100622]: from='client.73910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[100622]: from='client.73916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[100622]: from='client.73922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/286594434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/895202553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[102844]: from='client.73910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[102844]: from='client.73916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[102844]: from='client.73922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/286594434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/895202553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:37 smithi196 ceph-mon[76349]: from='client.73910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:37 smithi196 ceph-mon[76349]: from='client.73916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:37 smithi196 ceph-mon[76349]: from='client.73922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/286594434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/895202553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:38 smithi184 ceph-mon[100622]: from='client.63969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:38 smithi184 ceph-mon[100622]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:38 smithi184 ceph-mon[102844]: from='client.63969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:38 smithi184 ceph-mon[102844]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:38 smithi196 ceph-mon[76349]: from='client.63969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:38 smithi196 ceph-mon[76349]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:40 smithi184 ceph-mon[100622]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:40 smithi184 ceph-mon[102844]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:40 smithi196 ceph-mon[76349]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:41.698 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:41.410Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:41.699 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:41.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:41.699 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:41.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:45:43.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:42 smithi184 ceph-mon[100622]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:43.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:42 smithi184 ceph-mon[102844]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:42 smithi196 ceph-mon[76349]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:44 smithi184 ceph-mon[100622]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:44 smithi184 ceph-mon[102844]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:44 smithi196 ceph-mon[76349]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:46 smithi184 ceph-mon[100622]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:46 smithi184 ceph-mon[102844]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:46 smithi196 ceph-mon[76349]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:48 smithi184 ceph-mon[100622]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:48 smithi184 ceph-mon[102844]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:48 smithi196 ceph-mon[76349]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:50 smithi184 ceph-mon[100622]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:50 smithi184 ceph-mon[102844]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:50 smithi196 ceph-mon[76349]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:51.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:51.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:51.719 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:51.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:45:51.719 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:45:51.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:45:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:45:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:45:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:52 smithi184 ceph-mon[100622]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:52 smithi184 ceph-mon[102844]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:52 smithi196 ceph-mon[76349]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:54 smithi184 ceph-mon[100622]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:54 smithi184 ceph-mon[102844]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:54 smithi196 ceph-mon[76349]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:56 smithi196 ceph-mon[76349]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[100622]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[102844]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:45:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:58 smithi196 ceph-mon[76349]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:58 smithi184 ceph-mon[100622]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:58 smithi184 ceph-mon[102844]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:00 smithi196 ceph-mon[76349]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:00 smithi184 ceph-mon[100622]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:00 smithi184 ceph-mon[102844]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:01.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:01.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:01.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:46:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:02 smithi196 ceph-mon[76349]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:02 smithi184 ceph-mon[100622]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:02 smithi184 ceph-mon[102844]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:04 smithi196 ceph-mon[76349]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:04 smithi184 ceph-mon[100622]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:04 smithi184 ceph-mon[102844]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:06 smithi196 ceph-mon[76349]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:06 smithi184 ceph-mon[100622]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:06 smithi184 ceph-mon[102844]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:07.892 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 5h - - 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 13s ago 4h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 12s ago 4h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 13s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:46:08.416 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 13s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 13s ago 5h 486M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 13s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 13s ago 4h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 12s ago 4h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 13s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 13s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 13s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 13s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 4h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 4h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:46:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 4h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:46:08.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:46:08.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:46:08.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:46:08.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:46:08.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:46:08.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:08.727 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:46:08.727 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:46:08.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:46:08.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:46:08.997 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:46:08.997 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:46:08.998 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[100622]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[100622]: from='client.63981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1544178339' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[102844]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[102844]: from='client.63981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:08.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1544178339' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:08 smithi196 ceph-mon[76349]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:08 smithi196 ceph-mon[76349]: from='client.63981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1544178339' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:09.306 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:46:09.306 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:09.306 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:09.307 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:09.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:09 smithi196 ceph-mon[76349]: from='client.63987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:09 smithi196 ceph-mon[76349]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:09 smithi196 ceph-mon[76349]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2332593839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[100622]: from='client.63987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[100622]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[100622]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2332593839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[102844]: from='client.63987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[102844]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[102844]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2332593839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:10 smithi196 ceph-mon[76349]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:10 smithi184 ceph-mon[100622]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:10 smithi184 ceph-mon[102844]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:11.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:11.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:11.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:46:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:12 smithi196 ceph-mon[76349]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:12 smithi184 ceph-mon[100622]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:12 smithi184 ceph-mon[102844]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:15.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:14 smithi196 ceph-mon[76349]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:15.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:15.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:14 smithi184 ceph-mon[100622]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:15.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:15.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:14 smithi184 ceph-mon[102844]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:15.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:16.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[100622]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[102844]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:16 smithi196 ceph-mon[76349]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:18 smithi184 ceph-mon[100622]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:18 smithi184 ceph-mon[102844]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:18 smithi196 ceph-mon[76349]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:20 smithi184 ceph-mon[100622]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:20 smithi184 ceph-mon[102844]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:20 smithi196 ceph-mon[76349]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:21.414Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:21.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:21.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:46:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:22 smithi184 ceph-mon[100622]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:22 smithi184 ceph-mon[102844]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:22 smithi196 ceph-mon[76349]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:24 smithi184 ceph-mon[100622]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:24 smithi184 ceph-mon[102844]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:24 smithi196 ceph-mon[76349]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:26 smithi184 ceph-mon[100622]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:26 smithi184 ceph-mon[102844]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:26 smithi196 ceph-mon[76349]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:28 smithi184 ceph-mon[100622]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:28 smithi184 ceph-mon[102844]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:28 smithi196 ceph-mon[76349]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:30 smithi184 ceph-mon[100622]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:30 smithi184 ceph-mon[102844]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:30 smithi196 ceph-mon[76349]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:31.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:31.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:31.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:46:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:32 smithi184 ceph-mon[100622]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:32 smithi184 ceph-mon[102844]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:32 smithi196 ceph-mon[76349]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:35.055 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:34 smithi184 ceph-mon[100622]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:35.055 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:34 smithi184 ceph-mon[102844]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:34 smithi196 ceph-mon[76349]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:36.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.326 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.327 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[102844]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:36.327 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.327 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.327 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.327 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:36 smithi184 ceph-mon[100622]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:36 smithi196 ceph-mon[76349]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:38 smithi184 ceph-mon[100622]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:38 smithi184 ceph-mon[102844]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:38 smithi196 ceph-mon[76349]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:39.601 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:46:40.129 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 5s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 3s ago 4h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 5s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 5s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 5s ago 5h 487M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 3s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 5s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 5s ago 4h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 3s ago 4h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:46:40.130 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 5s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 5s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 5s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 5s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 3s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 3s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 3s ago 4h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 3s ago 4h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:46:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:46:40.438 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:46:40.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:46:40.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:46:40.440 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:46:40.711 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:46:40.711 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:46:40.711 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:46:40.711 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:46:40.712 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:46:40.713 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:46:40.713 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:46:40.713 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:46:40.713 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:46:41.019 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:46:41.019 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:41.019 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:41.019 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:41.019 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:41.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[102844]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:41.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[102844]: from='client.64017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[102844]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/341943912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:41.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[100622]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:41.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[100622]: from='client.64017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[100622]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/341943912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:40 smithi196 ceph-mon[76349]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:40 smithi196 ceph-mon[76349]: from='client.64017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:40 smithi196 ceph-mon[76349]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/341943912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:41.677 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:41.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:41.677 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:41.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:41.677 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:41.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[100622]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[100622]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3307651528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[102844]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[102844]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3307651528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:41 smithi196 ceph-mon[76349]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:41 smithi196 ceph-mon[76349]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3307651528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:46:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:42 smithi184 ceph-mon[100622]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:42 smithi184 ceph-mon[102844]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:42 smithi196 ceph-mon[76349]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:44 smithi184 ceph-mon[100622]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:44 smithi184 ceph-mon[102844]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:44 smithi196 ceph-mon[76349]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:46 smithi184 ceph-mon[100622]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:46 smithi184 ceph-mon[102844]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:47.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:46 smithi196 ceph-mon[76349]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:48 smithi184 ceph-mon[100622]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:48 smithi184 ceph-mon[102844]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:48 smithi196 ceph-mon[76349]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:50 smithi184 ceph-mon[100622]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:50 smithi184 ceph-mon[102844]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:50 smithi196 ceph-mon[76349]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:51.696 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:51.416Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:51.696 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:51.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:46:51.696 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:46:51.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:46:52.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:52.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:46:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:46:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:52 smithi184 ceph-mon[102844]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:52 smithi184 ceph-mon[100622]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:52 smithi196 ceph-mon[76349]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:54 smithi184 ceph-mon[102844]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:54 smithi184 ceph-mon[100622]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:54 smithi196 ceph-mon[76349]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:56.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.482 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:56 smithi196 ceph-mon[76349]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[100622]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:56 smithi184 ceph-mon[102844]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:46:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:58 smithi184 ceph-mon[100622]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:58 smithi184 ceph-mon[102844]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:58 smithi196 ceph-mon[76349]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:00 smithi196 ceph-mon[76349]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:00 smithi184 ceph-mon[100622]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:00 smithi184 ceph-mon[102844]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:01.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:01.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:01.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:47:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:02 smithi196 ceph-mon[76349]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:02 smithi184 ceph-mon[100622]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:02 smithi184 ceph-mon[102844]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:04 smithi196 ceph-mon[76349]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:04 smithi184 ceph-mon[100622]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:04 smithi184 ceph-mon[102844]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:06 smithi196 ceph-mon[76349]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:06 smithi184 ceph-mon[100622]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:06 smithi184 ceph-mon[102844]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:08 smithi184 ceph-mon[100622]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:08 smithi184 ceph-mon[102844]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:08 smithi196 ceph-mon[76349]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:10 smithi196 ceph-mon[76349]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:10 smithi184 ceph-mon[100622]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:10 smithi184 ceph-mon[102844]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:11.316 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:47:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:11.418Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:11.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:11.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:11.845 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:47:11.845 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 5h - - 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 16s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 15s ago 4h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 16s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 16s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 16s ago 5h 488M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 15s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 16s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 16s ago 4h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 15s ago 4h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 16s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:47:11.846 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 16s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 16s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 16s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 15s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 15s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 15s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 15s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 15s ago 4h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:47:11.847 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 15s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:47:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:47:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:47:12.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:47:12.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:12.156 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:12.157 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:47:12.424 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:47:12.425 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:47:12.732 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:47:12.732 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:12.733 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:13.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: from='client.74018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4084425960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1961675737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: from='client.74018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4084425960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:13.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1961675737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: from='client.74018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4084425960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1961675737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:13 smithi196 ceph-mon[76349]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:13 smithi184 ceph-mon[100622]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:13 smithi184 ceph-mon[102844]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:14 smithi196 ceph-mon[76349]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:14 smithi184 ceph-mon[100622]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:14 smithi184 ceph-mon[102844]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[100622]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:16 smithi184 ceph-mon[102844]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:16 smithi196 ceph-mon[76349]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:18.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:18 smithi184 ceph-mon[100622]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:18 smithi184 ceph-mon[102844]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:18 smithi196 ceph-mon[76349]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:20 smithi184 ceph-mon[102844]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:20 smithi184 ceph-mon[100622]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:20 smithi196 ceph-mon[76349]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:21.418Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:21.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:21.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:47:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:22 smithi184 ceph-mon[100622]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:22 smithi184 ceph-mon[102844]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:22 smithi196 ceph-mon[76349]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:24 smithi184 ceph-mon[100622]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:24 smithi184 ceph-mon[102844]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:24 smithi196 ceph-mon[76349]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:26 smithi184 ceph-mon[100622]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:26 smithi184 ceph-mon[102844]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:26 smithi196 ceph-mon[76349]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:28 smithi184 ceph-mon[100622]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:28 smithi184 ceph-mon[102844]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:28 smithi196 ceph-mon[76349]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:30 smithi184 ceph-mon[100622]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:30 smithi184 ceph-mon[102844]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:30 smithi196 ceph-mon[76349]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:31.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:31.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:31.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:47:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:32 smithi184 ceph-mon[100622]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:32 smithi184 ceph-mon[102844]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:32 smithi196 ceph-mon[76349]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:34 smithi184 ceph-mon[102844]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:34 smithi184 ceph-mon[100622]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:34 smithi196 ceph-mon[76349]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:36.752 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:36 smithi196 ceph-mon[76349]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:36.752 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.752 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.752 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[100622]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[102844]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:38.170 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:39.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:39 smithi184 ceph-mon[100622]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:39.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:39 smithi184 ceph-mon[102844]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:39 smithi196 ceph-mon[76349]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:40 smithi196 ceph-mon[76349]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:40.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:40 smithi184 ceph-mon[100622]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:40.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:40 smithi184 ceph-mon[102844]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:41.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:41.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:41.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:41.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:41.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:42 smithi184 ceph-mon[102844]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:42 smithi184 ceph-mon[100622]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:47:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:42 smithi196 ceph-mon[76349]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:43.030 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:47:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:43 smithi184 ceph-mon[100622]: from='client.74054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:43 smithi184 ceph-mon[102844]: from='client.74054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:43 smithi196 ceph-mon[76349]: from='client.74054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:43.556 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:47:43.556 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 8s ago 4h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (4h) 6s ago 4h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 8s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 8s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 8s ago 5h 488M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 6s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 8s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 8s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 6s ago 4h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 8s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 8s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 8s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:47:43.557 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 8s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 6s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 6s ago 5h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:47:43.558 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:47:43.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:47:43.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:47:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:47:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:47:43.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:47:43.868 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:47:44.142 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:47:44.143 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:47:44.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[100622]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[100622]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[100622]: from='client.74066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1704078996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[102844]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[102844]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[102844]: from='client.74066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1704078996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:44.145 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:44 smithi196 ceph-mon[76349]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:44 smithi196 ceph-mon[76349]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:44 smithi196 ceph-mon[76349]: from='client.74066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1704078996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:44.453 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:47:44.453 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:44.453 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:44.453 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:44.453 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:45 smithi196 ceph-mon[76349]: from='client.64113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3707157308' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:45 smithi184 ceph-mon[100622]: from='client.64113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3707157308' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:45 smithi184 ceph-mon[102844]: from='client.64113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3707157308' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:46 smithi196 ceph-mon[76349]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:46 smithi184 ceph-mon[100622]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:46 smithi184 ceph-mon[102844]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:48 smithi184 ceph-mon[102844]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:48 smithi184 ceph-mon[100622]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:48 smithi196 ceph-mon[76349]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:50 smithi184 ceph-mon[102844]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:50 smithi184 ceph-mon[100622]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:50 smithi196 ceph-mon[76349]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:51.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:51.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:47:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:47:51.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:47:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:47:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:47:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:52 smithi184 ceph-mon[102844]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:52 smithi184 ceph-mon[100622]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:52 smithi196 ceph-mon[76349]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:54 smithi184 ceph-mon[100622]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:54 smithi184 ceph-mon[102844]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:54 smithi196 ceph-mon[76349]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:56 smithi196 ceph-mon[76349]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[100622]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[102844]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:58 smithi196 ceph-mon[76349]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:58.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[100622]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:58.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:47:58.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:58 smithi184 ceph-mon[102844]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:59.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:00 smithi184 ceph-mon[100622]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:00 smithi184 ceph-mon[102844]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:00 smithi196 ceph-mon[76349]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:01.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:01.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:01.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:02 smithi184 ceph-mon[100622]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:48:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:02 smithi184 ceph-mon[102844]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:02 smithi196 ceph-mon[76349]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:04 smithi184 ceph-mon[100622]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:04 smithi184 ceph-mon[102844]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:04 smithi196 ceph-mon[76349]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:06 smithi184 ceph-mon[100622]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:06 smithi184 ceph-mon[102844]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:06 smithi196 ceph-mon[76349]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:08 smithi184 ceph-mon[100622]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:08 smithi184 ceph-mon[102844]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:08 smithi196 ceph-mon[76349]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:10 smithi184 ceph-mon[100622]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:10 smithi184 ceph-mon[102844]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:10 smithi196 ceph-mon[76349]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:11.422Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:11.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:11.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:48:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:12 smithi184 ceph-mon[100622]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:12 smithi184 ceph-mon[102844]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:12 smithi196 ceph-mon[76349]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:48:13.810673805Z level=info msg="Completed cleanup jobs" duration=11.223185ms 2024-09-21T00:48:14.750 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:48:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:14 smithi184 ceph-mon[100622]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:14 smithi184 ceph-mon[102844]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:14 smithi196 ceph-mon[76349]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:15.281 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:48:15.281 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 19s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 19s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 19s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 19s ago 5h 489M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 18s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 19s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:48:15.282 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (4h) 19s ago 4h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (4h) 18s ago 4h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 19s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 19s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 19s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 19s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 18s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 18s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 18s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 18s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 18s ago 5h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:48:15.283 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:48:15.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:48:15.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:48:15.284 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 18s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:48:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:15 smithi184 ceph-mon[100622]: from='client.74090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:15 smithi184 ceph-mon[100622]: from='client.74096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:15 smithi184 ceph-mon[102844]: from='client.74090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:15 smithi184 ceph-mon[102844]: from='client.74096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:48:15.597 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:48:15.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:15.598 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:48:15.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:48:15.598 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:48:15.598 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:48:15.878 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:48:15.878 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:48:15.879 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:48:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:15 smithi196 ceph-mon[76349]: from='client.74090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:15 smithi196 ceph-mon[76349]: from='client.74096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.190 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:48:16.191 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:16.191 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:16.191 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:16.191 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1927028805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='client.74114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1430307859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1927028805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='client.74114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1430307859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1927028805' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='client.74114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1430307859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:18.412 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[100622]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.413 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:18 smithi184 ceph-mon[102844]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:18 smithi196 ceph-mon[76349]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:20 smithi196 ceph-mon[76349]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:20 smithi184 ceph-mon[100622]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:20 smithi184 ceph-mon[102844]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:21.423Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:21.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:21.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:21.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:22.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:48:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:22 smithi184 ceph-mon[100622]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:22 smithi184 ceph-mon[102844]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:22 smithi196 ceph-mon[76349]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:24 smithi184 ceph-mon[100622]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:24 smithi184 ceph-mon[102844]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:24 smithi196 ceph-mon[76349]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:26 smithi184 ceph-mon[100622]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:26 smithi184 ceph-mon[102844]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:26 smithi196 ceph-mon[76349]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:28 smithi184 ceph-mon[100622]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:28 smithi184 ceph-mon[102844]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:28 smithi196 ceph-mon[76349]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:30.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:30 smithi184 ceph-mon[100622]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:30 smithi184 ceph-mon[102844]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:30 smithi196 ceph-mon[76349]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:31.767 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:31.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:31.767 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:31.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:31.767 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:31.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:32.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:32.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:48:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:32 smithi184 ceph-mon[100622]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:32 smithi184 ceph-mon[102844]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:32 smithi196 ceph-mon[76349]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:34 smithi184 ceph-mon[100622]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:34 smithi184 ceph-mon[102844]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:34 smithi196 ceph-mon[76349]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:36 smithi196 ceph-mon[76349]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[100622]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[102844]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:38 smithi196 ceph-mon[76349]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:38.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[100622]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[102844]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:40 smithi196 ceph-mon[76349]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:40 smithi184 ceph-mon[100622]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:40 smithi184 ceph-mon[102844]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:41.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:41.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:41.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:41.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:48:42.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:42 smithi184 ceph-mon[100622]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:42.733 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:42 smithi184 ceph-mon[102844]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:42 smithi196 ceph-mon[76349]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:44 smithi184 ceph-mon[102844]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:44 smithi184 ceph-mon[100622]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:44 smithi196 ceph-mon[76349]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:46.488 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:48:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:46 smithi184 ceph-mon[100622]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:46 smithi184 ceph-mon[102844]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:46 smithi196 ceph-mon[76349]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (4h) 11s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 5h 492M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 9s ago 5h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:48:47.016 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 9s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 9s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 9s ago 5h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:48:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 9s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:48:47.326 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:48:47.327 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:48:47.597 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:48:47.598 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[100622]: from='client.74126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[100622]: from='client.74132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[100622]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4008382039' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[102844]: from='client.74126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[102844]: from='client.74132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[102844]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4008382039' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:47 smithi196 ceph-mon[76349]: from='client.74126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:47 smithi196 ceph-mon[76349]: from='client.74132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:47 smithi196 ceph-mon[76349]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4008382039' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:47.909 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:48:47.909 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:47.909 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:47.909 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:47.909 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[102844]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[102844]: from='client.64185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4092395545' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[100622]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[100622]: from='client.64185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4092395545' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:48 smithi196 ceph-mon[76349]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:48 smithi196 ceph-mon[76349]: from='client.64185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4092395545' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:50 smithi184 ceph-mon[100622]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:50 smithi184 ceph-mon[102844]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:50 smithi196 ceph-mon[76349]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:51.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:51.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:48:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:48:51.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:48:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:48:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:48:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:52 smithi184 ceph-mon[100622]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:52 smithi184 ceph-mon[102844]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:52 smithi196 ceph-mon[76349]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:54 smithi184 ceph-mon[100622]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:54 smithi184 ceph-mon[102844]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:54 smithi196 ceph-mon[76349]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[100622]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[102844]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:56 smithi196 ceph-mon[76349]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:58 smithi196 ceph-mon[76349]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[102844]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[100622]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:48:59.857 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:59.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:00.768 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:00 smithi196 ceph-mon[76349]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:00 smithi184 ceph-mon[100622]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:00 smithi184 ceph-mon[102844]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:01.537 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:01.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:01.537 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:01.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:01.537 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:01.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:02.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:49:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:02 smithi196 ceph-mon[76349]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:02 smithi184 ceph-mon[100622]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:02 smithi184 ceph-mon[102844]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:04 smithi196 ceph-mon[76349]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:04 smithi184 ceph-mon[100622]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:04 smithi184 ceph-mon[102844]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:06 smithi196 ceph-mon[76349]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:06 smithi184 ceph-mon[100622]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:06 smithi184 ceph-mon[102844]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:08 smithi196 ceph-mon[76349]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:08 smithi184 ceph-mon[100622]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:08 smithi184 ceph-mon[102844]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:10 smithi184 ceph-mon[100622]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:10 smithi184 ceph-mon[102844]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:10 smithi196 ceph-mon[76349]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:11.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:11.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:11.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:49:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:12 smithi196 ceph-mon[76349]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:12 smithi184 ceph-mon[100622]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:12 smithi184 ceph-mon[102844]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:14 smithi196 ceph-mon[76349]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:14 smithi184 ceph-mon[100622]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:14 smithi184 ceph-mon[102844]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:16 smithi196 ceph-mon[76349]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[100622]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[102844]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:18.205 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 2s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:49:18.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 2s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 2s ago 5h 491M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 1s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 2s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 2s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 2s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 2s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 2s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 1s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 1s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:49:18.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 1s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 1s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 1s ago 5h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:49:18.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 1s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:49:19.039 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:49:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:49:19.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[102844]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[100622]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:18 smithi196 ceph-mon[76349]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:19.308 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:49:19.309 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:49:19.622 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:49:19.622 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:19.622 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:19.622 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:19.622 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:20.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:19 smithi196 ceph-mon[76349]: from='client.64197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:19 smithi196 ceph-mon[76349]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:19 smithi196 ceph-mon[76349]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3324270620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:20.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2953607320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[100622]: from='client.64197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[100622]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[100622]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3324270620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2953607320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[102844]: from='client.64197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[102844]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[102844]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3324270620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2953607320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:21.045 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:20 smithi196 ceph-mon[76349]: from='client.64221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:21.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:20 smithi196 ceph-mon[76349]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:20 smithi184 ceph-mon[100622]: from='client.64221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:20 smithi184 ceph-mon[100622]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:20 smithi184 ceph-mon[102844]: from='client.64221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:20 smithi184 ceph-mon[102844]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:21.428Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:21.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:21.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:21.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:22.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:49:22.764 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:22 smithi184 ceph-mon[100622]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:22.764 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:22 smithi184 ceph-mon[102844]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:22 smithi196 ceph-mon[76349]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:24 smithi184 ceph-mon[100622]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:24 smithi184 ceph-mon[102844]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:24 smithi196 ceph-mon[76349]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:26 smithi184 ceph-mon[100622]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:26 smithi184 ceph-mon[102844]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:26 smithi196 ceph-mon[76349]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:28 smithi184 ceph-mon[100622]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:28 smithi184 ceph-mon[102844]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:28 smithi196 ceph-mon[76349]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:30 smithi184 ceph-mon[100622]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:30 smithi184 ceph-mon[102844]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:30 smithi196 ceph-mon[76349]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:31.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:31.429Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:31.758 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:31.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:31.759 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:31.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:32.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:32.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:49:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:32 smithi184 ceph-mon[100622]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:32 smithi184 ceph-mon[102844]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:32 smithi196 ceph-mon[76349]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:34 smithi184 ceph-mon[100622]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:34 smithi184 ceph-mon[102844]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:34 smithi196 ceph-mon[76349]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[100622]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[102844]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:36 smithi196 ceph-mon[76349]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:38 smithi196 ceph-mon[76349]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[100622]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[102844]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:40 smithi196 ceph-mon[76349]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:41.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:40 smithi184 ceph-mon[100622]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:41.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:40 smithi184 ceph-mon[102844]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:41.430Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:41.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:41.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:42.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:42.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:42.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:42.130 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:49:43.043 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:42 smithi184 ceph-mon[100622]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:43.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:42 smithi184 ceph-mon[102844]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:42 smithi196 ceph-mon[76349]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:44 smithi196 ceph-mon[76349]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:44 smithi184 ceph-mon[100622]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:44 smithi184 ceph-mon[102844]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:46 smithi196 ceph-mon[76349]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:46 smithi184 ceph-mon[100622]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:46 smithi184 ceph-mon[102844]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:48 smithi196 ceph-mon[76349]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:48 smithi184 ceph-mon[100622]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:48 smithi184 ceph-mon[102844]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:49.922 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:49:50.446 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 12s ago 5h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 4h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 5h 492M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 12s ago 5h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:49:50.447 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 12s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 12s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 12s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 12s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 12s ago 5h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:49:50.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:49:50.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:49:50.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 4h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:49:50.757 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:49:50.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:49:50.757 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:49:50.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:50.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:49:50.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:49:50.759 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:49:51.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:49:51.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:49:51.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[100622]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:51.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[100622]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:51.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2973113378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:51.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[102844]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:51.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[102844]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:51.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2973113378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:50 smithi196 ceph-mon[76349]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:50 smithi196 ceph-mon[76349]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2973113378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:51.339 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:49:51.339 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:51.339 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:51.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:51.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:49:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:49:51.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:51 smithi196 ceph-mon[76349]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:51 smithi196 ceph-mon[76349]: from='client.64245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:51 smithi196 ceph-mon[76349]: from='client.64257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3984571730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:49:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:49:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[100622]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[100622]: from='client.64245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[100622]: from='client.64257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3984571730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[102844]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[102844]: from='client.64245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:52.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[102844]: from='client.64257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:52.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3984571730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:52 smithi196 ceph-mon[76349]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:52 smithi184 ceph-mon[100622]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:52 smithi184 ceph-mon[102844]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:54 smithi196 ceph-mon[76349]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:54 smithi184 ceph-mon[100622]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:54 smithi184 ceph-mon[102844]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:56 smithi196 ceph-mon[76349]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[102844]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[100622]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[102844]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[100622]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:58 smithi196 ceph-mon[76349]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:49:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:49:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:00 smithi184 ceph-mon[102844]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:00 smithi184 ceph-mon[100622]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:00 smithi196 ceph-mon[76349]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:01.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:01.432Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:01.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:01.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:02.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:02.037 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:50:02.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:02 smithi196 ceph-mon[76349]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:03.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:02 smithi184 ceph-mon[100622]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:03.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:02 smithi184 ceph-mon[102844]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:04 smithi184 ceph-mon[100622]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:04 smithi184 ceph-mon[102844]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:04 smithi196 ceph-mon[76349]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:06 smithi184 ceph-mon[100622]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:06 smithi184 ceph-mon[102844]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:06 smithi196 ceph-mon[76349]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:08 smithi184 ceph-mon[100622]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:08 smithi184 ceph-mon[102844]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:08 smithi196 ceph-mon[76349]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:10 smithi184 ceph-mon[100622]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:10 smithi184 ceph-mon[102844]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:10 smithi196 ceph-mon[76349]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:11.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:11.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:11.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:12.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:12.007 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:50:12.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:12.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:13 smithi184 ceph-mon[100622]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:13 smithi184 ceph-mon[102844]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:13 smithi196 ceph-mon[76349]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:15 smithi184 ceph-mon[100622]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:15 smithi184 ceph-mon[102844]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:15 smithi196 ceph-mon[76349]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[102844]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[100622]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:17 smithi196 ceph-mon[76349]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[102844]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[100622]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:19 smithi196 ceph-mon[76349]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:21 smithi184 ceph-mon[102844]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:21 smithi184 ceph-mon[100622]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:21 smithi196 ceph-mon[76349]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:21.633 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:50:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:21.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:21.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:21.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 5h - - 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 4s ago 5h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:50:22.165 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 5s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 5s ago 5h 493M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 5h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 5s ago 5h 350M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 5s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 4s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 5s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 5s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 5s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 5s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:50:22.166 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 5h 74.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 5s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:50:22.167 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:50:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:50:22.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:50:22.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:50:22.480 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:50:22.762 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:50:22.763 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:50:23.074 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:23.074 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:23.074 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:23 smithi196 ceph-mon[76349]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:23 smithi196 ceph-mon[76349]: from='client.74234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:23 smithi196 ceph-mon[76349]: from='client.74240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2474128481' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[100622]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[100622]: from='client.74234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[100622]: from='client.74240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2474128481' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[102844]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[102844]: from='client.74234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[102844]: from='client.74240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2474128481' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:24.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:24 smithi196 ceph-mon[76349]: from='client.64281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:24 smithi196 ceph-mon[76349]: from='client.64293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2586772246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:24.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[100622]: from='client.64281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[100622]: from='client.64293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2586772246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[102844]: from='client.64281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[102844]: from='client.64293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2586772246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:25 smithi184 ceph-mon[100622]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:25 smithi184 ceph-mon[102844]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:25 smithi196 ceph-mon[76349]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:27 smithi184 ceph-mon[100622]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:27 smithi184 ceph-mon[102844]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:27 smithi196 ceph-mon[76349]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:29 smithi184 ceph-mon[100622]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:29 smithi184 ceph-mon[102844]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:29 smithi196 ceph-mon[76349]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:31 smithi184 ceph-mon[102844]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:31 smithi184 ceph-mon[100622]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:31 smithi196 ceph-mon[76349]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:31.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:31.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:31.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:50:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:33 smithi196 ceph-mon[76349]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:33 smithi184 ceph-mon[100622]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:33 smithi184 ceph-mon[102844]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:35 smithi196 ceph-mon[76349]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:35 smithi184 ceph-mon[100622]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:35 smithi184 ceph-mon[102844]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:37 smithi196 ceph-mon[76349]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[100622]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[102844]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:39 smithi196 ceph-mon[76349]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[100622]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[102844]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:41 smithi196 ceph-mon[76349]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:41.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:41 smithi184 ceph-mon[100622]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:41.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:41 smithi184 ceph-mon[102844]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:41.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:41.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:41.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:50:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:43 smithi196 ceph-mon[76349]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:43.391 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:43 smithi184 ceph-mon[100622]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:43.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:43 smithi184 ceph-mon[102844]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:44.296 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:44.296 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:45.355 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:45 smithi196 ceph-mon[76349]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:45.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:45 smithi184 ceph-mon[100622]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:45.428 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:45 smithi184 ceph-mon[102844]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:47 smithi196 ceph-mon[76349]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:47 smithi184 ceph-mon[100622]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:47 smithi184 ceph-mon[102844]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:49 smithi196 ceph-mon[76349]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:49 smithi184 ceph-mon[100622]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:49 smithi184 ceph-mon[102844]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:51 smithi196 ceph-mon[76349]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:51.437 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:51 smithi184 ceph-mon[100622]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:51.437 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:51 smithi184 ceph-mon[102844]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:51.436Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:51.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:50:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:50:51.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:50:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:50:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:50:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:53.370 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:53 smithi184 ceph-mon[100622]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:53 smithi184 ceph-mon[102844]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.371 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:50:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:53 smithi196 ceph-mon[76349]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 15s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:50:53.896 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 17s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 17s ago 5h 494M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 15s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 17s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 15s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 17s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 17s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 17s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:50:53.897 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 17s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 15s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 15s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 15s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 15s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 15s ago 5h 74.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 17s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 15s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 17s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:50:53.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 15s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:50:54.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:50:54.480 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:50:54.481 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:50:54.481 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:50:54.481 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[100622]: from='client.74270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[100622]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[100622]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[100622]: from='client.74282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[102844]: from='client.74270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[102844]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:54.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[102844]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.482 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:54 smithi184 ceph-mon[102844]: from='client.74282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:54 smithi196 ceph-mon[76349]: from='client.74270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:54 smithi196 ceph-mon[76349]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:54 smithi196 ceph-mon[76349]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:54 smithi196 ceph-mon[76349]: from='client.74282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:54.790 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:54.790 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:54.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:54.790 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:54.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1400258572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[100622]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2277452579' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1400258572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[102844]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2277452579' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1400258572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:55 smithi196 ceph-mon[76349]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2277452579' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:56 smithi184 ceph-mon[100622]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:56 smithi184 ceph-mon[102844]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:56 smithi196 ceph-mon[76349]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:58.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[102844]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[100622]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:58 smithi196 ceph-mon[76349]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:50:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:00 smithi184 ceph-mon[100622]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:00 smithi184 ceph-mon[102844]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:00 smithi196 ceph-mon[76349]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:01.790 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:01.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:01.790 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:01.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:01.790 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:01.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:02.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:02.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:02.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:51:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:02 smithi184 ceph-mon[102844]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:02 smithi184 ceph-mon[100622]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:02 smithi196 ceph-mon[76349]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:04.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:04 smithi184 ceph-mon[100622]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:04.943 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:04 smithi184 ceph-mon[102844]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:05.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:04 smithi196 ceph-mon[76349]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:06 smithi196 ceph-mon[76349]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:06 smithi184 ceph-mon[100622]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:06 smithi184 ceph-mon[102844]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:08 smithi196 ceph-mon[76349]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:08 smithi184 ceph-mon[100622]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:08 smithi184 ceph-mon[102844]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:10 smithi184 ceph-mon[100622]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:10 smithi184 ceph-mon[102844]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:10 smithi196 ceph-mon[76349]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:11.438Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:11.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:11.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:51:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:12 smithi196 ceph-mon[76349]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:12 smithi184 ceph-mon[100622]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:12 smithi184 ceph-mon[102844]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:14 smithi196 ceph-mon[76349]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:14 smithi184 ceph-mon[102844]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:14 smithi184 ceph-mon[100622]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:16 smithi196 ceph-mon[76349]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:16 smithi184 ceph-mon[102844]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:16 smithi184 ceph-mon[100622]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.188 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:18 smithi196 ceph-mon[76349]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:19.188 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[102844]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[100622]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:20 smithi184 ceph-mon[100622]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:20 smithi184 ceph-mon[102844]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:20 smithi196 ceph-mon[76349]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:21.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:21.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:21.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:51:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:22 smithi184 ceph-mon[100622]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:22 smithi184 ceph-mon[102844]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:22 smithi196 ceph-mon[76349]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:25.092 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:51:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:24 smithi196 ceph-mon[76349]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:24 smithi184 ceph-mon[100622]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:24 smithi184 ceph-mon[102844]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 5h - - 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 8s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 7s ago 5h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 8s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:51:25.627 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 8s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 8s ago 5h 495M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 8s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 8s ago 5h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 7s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 8s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 8s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 8s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 8s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:51:25.628 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 7s ago 5h 76.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:51:25.629 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 8s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:51:25.629 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:51:25.629 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 8s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:51:25.629 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:51:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:51:25.965 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:51:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:25 smithi196 ceph-mon[76349]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:26.159 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:25 smithi184 ceph-mon[100622]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:26.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:25 smithi184 ceph-mon[102844]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:51:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:51:26.246 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:51:26.597 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:51:26.597 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:26.597 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:27.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[102844]: from='client.74312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[102844]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:27.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[102844]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1130457045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:27.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/248556631' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:27.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[100622]: from='client.74312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[100622]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:27.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[100622]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1130457045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:27.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/248556631' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:26 smithi196 ceph-mon[76349]: from='client.74312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:26 smithi196 ceph-mon[76349]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:26 smithi196 ceph-mon[76349]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1130457045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/248556631' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:27 smithi184 ceph-mon[100622]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:27 smithi184 ceph-mon[102844]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:27 smithi196 ceph-mon[76349]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:28 smithi184 ceph-mon[100622]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:28 smithi184 ceph-mon[102844]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:28 smithi196 ceph-mon[76349]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:30 smithi184 ceph-mon[100622]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:30 smithi184 ceph-mon[102844]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:30 smithi196 ceph-mon[76349]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:31.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:31.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:31.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:51:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:32 smithi184 ceph-mon[102844]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:32 smithi184 ceph-mon[100622]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:32 smithi196 ceph-mon[76349]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:34 smithi184 ceph-mon[100622]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:34 smithi184 ceph-mon[102844]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:34 smithi196 ceph-mon[76349]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:36 smithi184 ceph-mon[100622]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:36 smithi184 ceph-mon[102844]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:36 smithi196 ceph-mon[76349]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:39 smithi196 ceph-mon[76349]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[102844]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[100622]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:41 smithi196 ceph-mon[76349]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:41.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:41 smithi184 ceph-mon[100622]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:41.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:41 smithi184 ceph-mon[102844]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:41.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:41.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:41.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:51:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:43 smithi184 ceph-mon[100622]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:43 smithi184 ceph-mon[102844]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:43 smithi196 ceph-mon[76349]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:45.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:45 smithi196 ceph-mon[76349]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:45.438 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:45 smithi184 ceph-mon[100622]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:45.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:45 smithi184 ceph-mon[102844]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:47 smithi196 ceph-mon[76349]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:47 smithi184 ceph-mon[100622]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:47 smithi184 ceph-mon[102844]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:49 smithi196 ceph-mon[76349]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:49 smithi184 ceph-mon[100622]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:49 smithi184 ceph-mon[102844]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:51 smithi196 ceph-mon[76349]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:51.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:51 smithi184 ceph-mon[100622]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:51.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:51 smithi184 ceph-mon[102844]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:51.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:51.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:51:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:51:51.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:51:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:51:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:51:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:53 smithi196 ceph-mon[76349]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:53 smithi184 ceph-mon[100622]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:53 smithi184 ceph-mon[102844]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:55 smithi196 ceph-mon[76349]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:55 smithi184 ceph-mon[100622]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:55 smithi184 ceph-mon[102844]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:56.892 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:51:57.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:57 smithi184 ceph-mon[100622]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:57.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:57 smithi184 ceph-mon[102844]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:57 smithi196 ceph-mon[76349]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:51:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 0s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 5h 494M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 18s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 18s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 18s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 18s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 18s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:51:57.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 18s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:51:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 18s ago 5h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:51:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:51:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:51:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:51:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 18s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:51:57.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:51:57.735 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:51:58.008 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:51:58.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:51:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[100622]: from='client.64377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1646199157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[102844]: from='client.64377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1646199157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:58.320 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:51:58.320 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:58.320 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:58.320 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:58.320 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:58 smithi196 ceph-mon[76349]: from='client.64377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1646199157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='client.74348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='client.64389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='client.74366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3080383289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='client.74348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='client.64389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='client.74366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3080383289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='client.74348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='client.64389 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='client.74366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3080383289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:51:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:00 smithi184 ceph-mon[100622]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:00 smithi184 ceph-mon[102844]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:00 smithi196 ceph-mon[76349]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:01.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:01.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:01.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:01.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:01.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:02 smithi184 ceph-mon[100622]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:02 smithi184 ceph-mon[102844]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:52:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:02 smithi196 ceph-mon[76349]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:04 smithi196 ceph-mon[76349]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:04 smithi184 ceph-mon[100622]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:04 smithi184 ceph-mon[102844]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:06.753 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:06 smithi184 ceph-mon[100622]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:06.753 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:06 smithi184 ceph-mon[102844]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:06.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:06 smithi196 ceph-mon[76349]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:08.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:08 smithi184 ceph-mon[100622]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:08.650 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:08 smithi184 ceph-mon[102844]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:08 smithi196 ceph-mon[76349]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:10 smithi196 ceph-mon[76349]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:10 smithi184 ceph-mon[100622]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:10 smithi184 ceph-mon[102844]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:11.441Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:11.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:11.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:52:12.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:12 smithi196 ceph-mon[76349]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:12.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:12 smithi184 ceph-mon[100622]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:12.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:12 smithi184 ceph-mon[102844]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:14 smithi196 ceph-mon[76349]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:14 smithi184 ceph-mon[100622]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:14 smithi184 ceph-mon[102844]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:16 smithi196 ceph-mon[76349]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:16 smithi184 ceph-mon[100622]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:16 smithi184 ceph-mon[102844]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:18 smithi196 ceph-mon[76349]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[100622]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:18 smithi184 ceph-mon[102844]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:20.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:20 smithi184 ceph-mon[100622]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:20 smithi184 ceph-mon[102844]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:20 smithi196 ceph-mon[76349]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:21.441Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:21.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:21.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T00:52:22.375 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:22 smithi184 ceph-mon[100622]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:22 smithi184 ceph-mon[102844]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:22 smithi196 ceph-mon[76349]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:24 smithi184 ceph-mon[100622]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:24 smithi184 ceph-mon[102844]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:24 smithi196 ceph-mon[76349]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:26 smithi184 ceph-mon[100622]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:27.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:26 smithi184 ceph-mon[102844]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:27.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:26 smithi196 ceph-mon[76349]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:28.613 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:52:29.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:28 smithi196 ceph-mon[76349]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:29.154 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:28 smithi184 ceph-mon[100622]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:29.155 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:28 smithi184 ceph-mon[102844]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:29.161 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:52:29.161 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T00:52:29.161 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T00:52:29.161 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 11s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:52:29.161 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 11s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 11s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 11s ago 5h 495M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 11s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:52:29.162 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 11s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 11s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 11s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 11s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:52:29.163 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:52:29.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:52:29.164 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 10s ago 5h 77.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:52:29.164 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 11s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:52:29.164 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:52:29.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 11s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:52:29.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:52:29.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:52:29.478 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:52:29.754 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:52:29.755 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:52:29.755 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:52:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[102844]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[102844]: from='client.64419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3609612103' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[100622]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[100622]: from='client.64419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3609612103' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:30.077 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:52:30.077 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:30.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:30.077 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:30.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:29 smithi196 ceph-mon[76349]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:29 smithi196 ceph-mon[76349]: from='client.64419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3609612103' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:30.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[102844]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[102844]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:30.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[102844]: from='client.74402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2319273041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:30.996 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[100622]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[100622]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:30.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[100622]: from='client.74402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:30.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2319273041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:30 smithi196 ceph-mon[76349]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:30 smithi196 ceph-mon[76349]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:30 smithi196 ceph-mon[76349]: from='client.74402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2319273041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:31.442Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:31.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:31.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:52:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:32 smithi184 ceph-mon[102844]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:32 smithi184 ceph-mon[100622]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:32 smithi196 ceph-mon[76349]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:34 smithi184 ceph-mon[100622]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:34 smithi184 ceph-mon[102844]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:34 smithi196 ceph-mon[76349]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:36 smithi184 ceph-mon[100622]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:36 smithi184 ceph-mon[102844]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:36 smithi196 ceph-mon[76349]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[100622]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:38 smithi184 ceph-mon[102844]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:38 smithi196 ceph-mon[76349]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:41 smithi184 ceph-mon[100622]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:41 smithi184 ceph-mon[102844]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:41 smithi196 ceph-mon[76349]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:41.442Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:41.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:41.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:52:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:43 smithi184 ceph-mon[100622]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:43 smithi184 ceph-mon[102844]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:43 smithi196 ceph-mon[76349]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:45 smithi196 ceph-mon[76349]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:45 smithi184 ceph-mon[100622]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:45 smithi184 ceph-mon[102844]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:47 smithi196 ceph-mon[76349]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:47 smithi184 ceph-mon[100622]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:47 smithi184 ceph-mon[102844]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:49 smithi184 ceph-mon[100622]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:49 smithi184 ceph-mon[102844]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:49.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:49 smithi196 ceph-mon[76349]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:51 smithi196 ceph-mon[76349]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:51.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:51 smithi184 ceph-mon[100622]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:51.444 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:51 smithi184 ceph-mon[102844]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:51.443Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:51.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:52:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:52:51.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:52:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:52:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:52:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:53 smithi196 ceph-mon[76349]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:53 smithi184 ceph-mon[100622]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:53 smithi184 ceph-mon[102844]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:55 smithi196 ceph-mon[76349]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:55 smithi184 ceph-mon[100622]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:55 smithi184 ceph-mon[102844]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:57 smithi196 ceph-mon[76349]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:57 smithi184 ceph-mon[100622]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:57 smithi184 ceph-mon[102844]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:58 smithi196 ceph-mon[76349]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[100622]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[102844]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:00.373 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:53:00.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:00 smithi184 ceph-mon[100622]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:00.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:00 smithi184 ceph-mon[102844]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:00 smithi196 ceph-mon[76349]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:00.900 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:53:00.900 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T00:53:00.900 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T00:53:00.900 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:53:00.900 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 3s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 3s ago 5h 496M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 1s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 3s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 1s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 1s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:53:00.901 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 1s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 1s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 1s ago 5h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 1s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:53:00.902 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 1s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:53:01.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:53:01.213 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:53:01.482 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:53:01.482 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:53:01.482 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:53:01.483 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:53:01.483 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:01.444Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:01.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:01.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:01.484 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:01.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[100622]: from='client.64449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[100622]: from='client.74420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[100622]: from='client.74426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3735296732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[102844]: from='client.64449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[102844]: from='client.74420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[102844]: from='client.74426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3735296732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:01.798 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:53:01.798 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:01.798 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:01.798 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:01.799 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:01 smithi196 ceph-mon[76349]: from='client.64449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:01 smithi196 ceph-mon[76349]: from='client.74420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:01 smithi196 ceph-mon[76349]: from='client.74426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3735296732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:53:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:02 smithi196 ceph-mon[76349]: from='client.64473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:02 smithi196 ceph-mon[76349]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3206629201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[102844]: from='client.64473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[102844]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3206629201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[100622]: from='client.64473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[100622]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3206629201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:04 smithi196 ceph-mon[76349]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:04 smithi184 ceph-mon[102844]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:04 smithi184 ceph-mon[100622]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:06 smithi196 ceph-mon[76349]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:06 smithi184 ceph-mon[100622]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:06 smithi184 ceph-mon[102844]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:08 smithi196 ceph-mon[76349]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:08 smithi184 ceph-mon[100622]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:08 smithi184 ceph-mon[102844]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:10.858 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:10 smithi184 ceph-mon[100622]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:10.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:10 smithi184 ceph-mon[102844]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:10 smithi196 ceph-mon[76349]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:11.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:11.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:11.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:12.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:53:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:12 smithi196 ceph-mon[76349]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:12 smithi184 ceph-mon[100622]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:12 smithi184 ceph-mon[102844]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:14 smithi196 ceph-mon[76349]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:14 smithi184 ceph-mon[100622]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:14 smithi184 ceph-mon[102844]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:16 smithi196 ceph-mon[76349]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:16 smithi184 ceph-mon[100622]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:16 smithi184 ceph-mon[102844]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[100622]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[102844]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:18 smithi196 ceph-mon[76349]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[100622]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:20 smithi184 ceph-mon[102844]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:20 smithi196 ceph-mon[76349]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:21.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:21.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:21.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:53:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:22 smithi196 ceph-mon[76349]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:22 smithi184 ceph-mon[100622]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:22 smithi184 ceph-mon[102844]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:24 smithi196 ceph-mon[76349]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:24 smithi184 ceph-mon[100622]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:24 smithi184 ceph-mon[102844]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:26 smithi196 ceph-mon[76349]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:26 smithi184 ceph-mon[100622]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:26 smithi184 ceph-mon[102844]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:28.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:28 smithi196 ceph-mon[76349]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:28.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:28 smithi184 ceph-mon[100622]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:28.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:28 smithi184 ceph-mon[102844]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:30.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:30 smithi184 ceph-mon[100622]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:30.789 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:30 smithi184 ceph-mon[102844]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:30 smithi196 ceph-mon[76349]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:31.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:31.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:31.747 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:31.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:31.747 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:31.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:31.747 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:31.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:32.096 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:53:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:53:32.638 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 14s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 14s ago 5h 498M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 13s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 14s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 14s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 14s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 14s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 14s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 5h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 13s ago 5h 78.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:53:32.639 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:53:32.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:53:32.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:53:32.640 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 5h 91.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:53:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:32 smithi196 ceph-mon[76349]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:32 smithi196 ceph-mon[76349]: from='client.74450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:32.964 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:53:32.965 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:53:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:53:32.966 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:53:32.966 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:53:32.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:32 smithi184 ceph-mon[100622]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:32.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:32 smithi184 ceph-mon[100622]: from='client.74450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:32.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:32 smithi184 ceph-mon[102844]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:32.966 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:32 smithi184 ceph-mon[102844]: from='client.74450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.245 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:53:33.246 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:53:33.567 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:53:33.567 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:33.567 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:33.567 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:33.567 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[100622]: from='client.74456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[100622]: from='client.64497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3781670917' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3094889425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[102844]: from='client.74456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[102844]: from='client.64497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3781670917' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3094889425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:33 smithi196 ceph-mon[76349]: from='client.74456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:33 smithi196 ceph-mon[76349]: from='client.64497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3781670917' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3094889425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:34 smithi196 ceph-mon[76349]: from='client.64509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:34 smithi196 ceph-mon[76349]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:34 smithi184 ceph-mon[100622]: from='client.64509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:34 smithi184 ceph-mon[100622]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:34 smithi184 ceph-mon[102844]: from='client.64509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:34 smithi184 ceph-mon[102844]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:36 smithi196 ceph-mon[76349]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:36 smithi184 ceph-mon[100622]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:36 smithi184 ceph-mon[102844]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:38 smithi196 ceph-mon[76349]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[100622]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[102844]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:40 smithi196 ceph-mon[76349]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[100622]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:40 smithi184 ceph-mon[102844]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:41.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:41.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:41.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:41.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:53:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:42 smithi196 ceph-mon[76349]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:42 smithi184 ceph-mon[100622]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:42 smithi184 ceph-mon[102844]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:44 smithi196 ceph-mon[76349]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:44 smithi184 ceph-mon[100622]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:44 smithi184 ceph-mon[102844]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:46 smithi196 ceph-mon[76349]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:46 smithi184 ceph-mon[100622]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:46 smithi184 ceph-mon[102844]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:48.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:48 smithi196 ceph-mon[76349]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:48 smithi184 ceph-mon[100622]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:48 smithi184 ceph-mon[102844]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:50 smithi196 ceph-mon[76349]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:50.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:50 smithi184 ceph-mon[100622]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:50.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:50 smithi184 ceph-mon[102844]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:51.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:51.447Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:51.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:51.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:53:51.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:53:51.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:53:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:52.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:53:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:53:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:52 smithi196 ceph-mon[76349]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:52.961 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:52 smithi184 ceph-mon[100622]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:52.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:52 smithi184 ceph-mon[102844]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:54 smithi196 ceph-mon[76349]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:54 smithi184 ceph-mon[100622]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:54 smithi184 ceph-mon[102844]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:56 smithi196 ceph-mon[76349]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:56 smithi184 ceph-mon[100622]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:56 smithi184 ceph-mon[102844]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[100622]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[102844]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:58 smithi196 ceph-mon[76349]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:53:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:00 smithi184 ceph-mon[100622]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:00 smithi184 ceph-mon[102844]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:00 smithi196 ceph-mon[76349]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:01.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:01.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:01.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:54:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:02 smithi184 ceph-mon[100622]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:02 smithi184 ceph-mon[102844]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:02 smithi196 ceph-mon[76349]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:03.864 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:54:04.393 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:54:04.393 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T00:54:04.393 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 5h - - 2024-09-21T00:54:04.393 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 4s ago 5h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 4s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4h) 6s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 6s ago 5h 497M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 4s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 6s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 4s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 6s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 6s ago 5h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 6s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 6s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:54:04.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 4s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 4s ago 5h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 4s ago 5h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 4s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 4s ago 5h 78.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 6s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 4s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 6s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:54:04.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 4s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:54:04.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:54:04.706 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:54:04.982 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:54:04.983 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:54:04.983 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[100622]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[100622]: from='client.64521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[100622]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/163672164' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[102844]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[102844]: from='client.64521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[102844]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:04.984 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/163672164' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:05.294 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:54:05.294 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:05.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:05.295 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:05.295 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:04 smithi196 ceph-mon[76349]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:04 smithi196 ceph-mon[76349]: from='client.64521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:04 smithi196 ceph-mon[76349]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/163672164' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[100622]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[100622]: from='client.64545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/86876620' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[102844]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[102844]: from='client.64545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/86876620' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:05 smithi196 ceph-mon[76349]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:05 smithi196 ceph-mon[76349]: from='client.64545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/86876620' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:06 smithi184 ceph-mon[100622]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:06 smithi184 ceph-mon[102844]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:06 smithi196 ceph-mon[76349]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:08 smithi184 ceph-mon[100622]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:54:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:08 smithi184 ceph-mon[102844]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:54:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:08 smithi196 ceph-mon[76349]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T00:54:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:10 smithi184 ceph-mon[100622]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:10 smithi184 ceph-mon[102844]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:11.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:10 smithi196 ceph-mon[76349]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:11.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:11.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:11.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:11.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:11.718 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:11.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:54:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:12 smithi184 ceph-mon[100622]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:12 smithi184 ceph-mon[102844]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:12 smithi196 ceph-mon[76349]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:54:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:14 smithi184 ceph-mon[100622]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:54:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:14 smithi184 ceph-mon[102844]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:54:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:14 smithi196 ceph-mon[76349]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:54:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:16 smithi184 ceph-mon[100622]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:54:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:16 smithi184 ceph-mon[102844]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:54:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:16 smithi196 ceph-mon[76349]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:54:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:19 smithi196 ceph-mon[76349]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T00:54:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[100622]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T00:54:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[102844]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[100622]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:54:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[102844]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:54:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:20 smithi196 ceph-mon[76349]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T00:54:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:21.449Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:21.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:21.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:54:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:22 smithi184 ceph-mon[100622]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:54:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:22 smithi184 ceph-mon[102844]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:54:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:22 smithi196 ceph-mon[76349]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:54:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:24 smithi184 ceph-mon[100622]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:54:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:24 smithi184 ceph-mon[102844]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:54:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:24 smithi196 ceph-mon[76349]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:54:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:26 smithi184 ceph-mon[100622]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 335 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:26 smithi184 ceph-mon[102844]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 335 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:26 smithi196 ceph-mon[76349]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 335 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:28 smithi184 ceph-mon[100622]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 339 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:54:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:28 smithi184 ceph-mon[102844]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 339 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:54:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:28 smithi196 ceph-mon[76349]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 339 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:54:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:30 smithi196 ceph-mon[76349]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 343 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:54:31.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:30 smithi184 ceph-mon[100622]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 343 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:54:31.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:30 smithi184 ceph-mon[102844]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 343 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:54:31.461 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:31.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:31.461 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:31.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:31.461 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:31.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:32.241 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:32.241 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:32.241 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:54:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:32 smithi184 ceph-mon[100622]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:54:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:32 smithi184 ceph-mon[102844]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:54:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:32 smithi196 ceph-mon[76349]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 327 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T00:54:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:34 smithi196 ceph-mon[76349]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T00:54:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:34 smithi184 ceph-mon[100622]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T00:54:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:34 smithi184 ceph-mon[102844]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T00:54:35.588 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:54:36.117 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (4h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 18s ago 5h 497M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 16s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 18s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:54:36.118 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 18s ago 5h 89.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 5h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 5h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 5h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 16s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 16s ago 5h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 16s ago 5h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 16s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (4h) 16s ago 5h 78.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 16s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:54:36.119 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:54:36.425 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:54:36.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:54:36.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:54:36.427 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:54:36.697 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:54:36.698 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:54:36.699 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:54:36.699 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:54:36.699 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:54:37.005 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:54:37.005 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:37.005 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:37.005 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:37.005 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:37.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[100622]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[100622]: from='client.64557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[100622]: from='client.74528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[100622]: from='client.64569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2919531277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[102844]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[102844]: from='client.64557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[102844]: from='client.74528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[102844]: from='client.64569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2919531277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:36 smithi196 ceph-mon[76349]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:54:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:36 smithi196 ceph-mon[76349]: from='client.64557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:36 smithi196 ceph-mon[76349]: from='client.74528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:36 smithi196 ceph-mon[76349]: from='client.64569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2919531277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:37 smithi196 ceph-mon[76349]: from='client.74546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/869664041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:38.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:37 smithi184 ceph-mon[100622]: from='client.74546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:38.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/869664041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:38.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:37 smithi184 ceph-mon[102844]: from='client.74546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:38.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/869664041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:38 smithi196 ceph-mon[76349]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[100622]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[102844]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[100622]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[102844]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:40 smithi196 ceph-mon[76349]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:54:41.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:41.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:41.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:41.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:54:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:42 smithi196 ceph-mon[76349]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:54:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:42 smithi184 ceph-mon[100622]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:54:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:42 smithi184 ceph-mon[102844]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T00:54:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:44 smithi196 ceph-mon[76349]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:54:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:44 smithi184 ceph-mon[100622]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:54:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:44 smithi184 ceph-mon[102844]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:54:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:46 smithi196 ceph-mon[76349]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-21T00:54:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:46 smithi184 ceph-mon[100622]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-21T00:54:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:46 smithi184 ceph-mon[102844]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-21T00:54:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:48 smithi196 ceph-mon[76349]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T00:54:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:48 smithi184 ceph-mon[100622]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T00:54:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:48 smithi184 ceph-mon[102844]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T00:54:51.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:50 smithi196 ceph-mon[76349]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:50 smithi184 ceph-mon[100622]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:50 smithi184 ceph-mon[102844]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T00:54:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:51.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:51.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:54:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:54:51.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:54:52.070 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:52.073 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:52.073 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:52.073 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:54:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:54:53.069 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:52 smithi184 ceph-mon[100622]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:53.069 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:52 smithi184 ceph-mon[102844]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:52 smithi196 ceph-mon[76349]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:54 smithi196 ceph-mon[76349]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:54 smithi184 ceph-mon[100622]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:54 smithi184 ceph-mon[102844]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:56 smithi196 ceph-mon[76349]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:56 smithi184 ceph-mon[100622]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:56 smithi184 ceph-mon[102844]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:58 smithi196 ceph-mon[76349]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[100622]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[102844]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:54:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:54:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:54:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:54:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[100622]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[102844]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:00 smithi196 ceph-mon[76349]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:01.452Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:01.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:01.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:55:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:02 smithi184 ceph-mon[100622]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:02 smithi184 ceph-mon[102844]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:02 smithi196 ceph-mon[76349]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:04 smithi184 ceph-mon[100622]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:04 smithi184 ceph-mon[102844]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:04 smithi196 ceph-mon[76349]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:06 smithi184 ceph-mon[100622]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:06 smithi184 ceph-mon[102844]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:55:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:06 smithi196 ceph-mon[76349]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 5h - - 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 5h - - 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 7s ago 5h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (4h) 7s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:55:07.826 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 9s ago 5h 499M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 7s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 9s ago 5h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 7s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 9s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 9s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 9s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 9s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 7s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 7s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 7s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 7s ago 5h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 7s ago 5h 79.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:55:07.827 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 9s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:55:07.828 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:55:07.828 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 9s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:55:07.828 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 7s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:55:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:55:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:55:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:55:08.142 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:55:08.409 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:55:08.409 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:55:08.409 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:55:08.410 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:55:08.718 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:55:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:08.719 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:08.719 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: from='client.64593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: from='client.74564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2698522873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3166307141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: from='client.64593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: from='client.74564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2698522873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3166307141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: from='client.64593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: from='client.74564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2698522873' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3166307141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:10.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:09 smithi196 ceph-mon[76349]: from='client.74582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:09 smithi184 ceph-mon[100622]: from='client.74582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:10.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:09 smithi184 ceph-mon[102844]: from='client.74582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:10 smithi184 ceph-mon[100622]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:10 smithi184 ceph-mon[102844]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:10 smithi196 ceph-mon[76349]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:11.452Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:11.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:11.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:55:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:12.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:13.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:12 smithi196 ceph-mon[76349]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:12 smithi184 ceph-mon[100622]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:12 smithi184 ceph-mon[102844]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:15.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:14 smithi184 ceph-mon[100622]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:15.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:15.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:14 smithi184 ceph-mon[102844]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:15.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:14 smithi196 ceph-mon[76349]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:16 smithi184 ceph-mon[100622]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:16 smithi184 ceph-mon[102844]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:16 smithi196 ceph-mon[76349]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[102844]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[100622]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:18 smithi196 ceph-mon[76349]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[100622]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[102844]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:21 smithi196 ceph-mon[76349]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:21.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:21.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:21.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:55:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:23 smithi196 ceph-mon[76349]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:23 smithi184 ceph-mon[100622]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:23 smithi184 ceph-mon[102844]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:25 smithi196 ceph-mon[76349]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:25 smithi184 ceph-mon[100622]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:25 smithi184 ceph-mon[102844]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:27 smithi196 ceph-mon[76349]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:27 smithi184 ceph-mon[100622]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:27 smithi184 ceph-mon[102844]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:29 smithi196 ceph-mon[76349]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:29 smithi184 ceph-mon[100622]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:29 smithi184 ceph-mon[102844]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:31 smithi184 ceph-mon[100622]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:31 smithi184 ceph-mon[102844]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:31 smithi196 ceph-mon[76349]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:31.454Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:31.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:31.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:55:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:33.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:33 smithi184 ceph-mon[100622]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:33.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:33 smithi184 ceph-mon[102844]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:33 smithi196 ceph-mon[76349]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:35 smithi196 ceph-mon[76349]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:35 smithi184 ceph-mon[100622]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:35 smithi184 ceph-mon[102844]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:37 smithi196 ceph-mon[76349]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:37 smithi184 ceph-mon[100622]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:37 smithi184 ceph-mon[102844]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:39.004 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:55:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[102844]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[100622]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:39 smithi196 ceph-mon[76349]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:55:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T00:55:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 5h - - 2024-09-21T00:55:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:55:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4h) 0s ago 5h 500M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 19s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 0s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 0s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 0s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 0s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 0s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 19s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:55:39.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 19s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 19s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 19s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 79.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 19s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:55:39.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:55:39.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:55:39.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:55:39.846 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:55:39.846 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:55:39.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:55:39.846 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:55:40.119 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:55:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:55:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:55:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:55:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:40 smithi196 ceph-mon[76349]: from='client.74594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3801404718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:40.428 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:55:40.428 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:40.428 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:40.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:40 smithi184 ceph-mon[100622]: from='client.74594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:40.428 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3801404718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:40.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:40 smithi184 ceph-mon[102844]: from='client.74594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:40.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3801404718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='client.74600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='client.74606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='client.64653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3211131476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='client.74600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='client.74606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='client.64653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3211131476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='client.74600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='client.74606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='client.64653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:55:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3211131476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:41.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:41.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:41.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:55:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:43 smithi196 ceph-mon[76349]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:43 smithi184 ceph-mon[100622]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:43 smithi184 ceph-mon[102844]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:45 smithi196 ceph-mon[76349]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:45 smithi184 ceph-mon[100622]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:45 smithi184 ceph-mon[102844]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:47 smithi196 ceph-mon[76349]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:47 smithi184 ceph-mon[100622]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:47 smithi184 ceph-mon[102844]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:49 smithi196 ceph-mon[76349]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:49 smithi184 ceph-mon[100622]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:49.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:49 smithi184 ceph-mon[102844]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:51 smithi184 ceph-mon[100622]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:51 smithi184 ceph-mon[102844]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:51 smithi196 ceph-mon[76349]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:51.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:51.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:55:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:55:51.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:55:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:52.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:55:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:55:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:53 smithi196 ceph-mon[76349]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:53 smithi184 ceph-mon[100622]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:53 smithi184 ceph-mon[102844]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:55.454 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:55 smithi184 ceph-mon[100622]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:55.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:55 smithi184 ceph-mon[102844]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:55.521 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:55 smithi196 ceph-mon[76349]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:57 smithi196 ceph-mon[76349]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:57 smithi184 ceph-mon[100622]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:57 smithi184 ceph-mon[102844]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:58 smithi184 ceph-mon[100622]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:58 smithi184 ceph-mon[102844]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:58 smithi196 ceph-mon[76349]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[100622]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[102844]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:00 smithi196 ceph-mon[76349]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:01.738 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:01.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:01.738 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:01.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:01.738 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:01.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:56:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:02 smithi184 ceph-mon[100622]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:02 smithi184 ceph-mon[102844]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:02 smithi196 ceph-mon[76349]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:04 smithi184 ceph-mon[100622]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:04 smithi184 ceph-mon[102844]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:04 smithi196 ceph-mon[76349]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:07.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:06 smithi196 ceph-mon[76349]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:06 smithi184 ceph-mon[100622]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:06 smithi184 ceph-mon[102844]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:08 smithi184 ceph-mon[100622]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:08 smithi184 ceph-mon[102844]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:08 smithi196 ceph-mon[76349]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:56:10.714 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:56:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:10 smithi184 ceph-mon[100622]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:10 smithi184 ceph-mon[102844]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:10 smithi196 ceph-mon[76349]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 5h - - 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:56:11.245 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 5h 500M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (4h) 10s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (4h) 12s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 12s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 12s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 12s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 12s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 10s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 10s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 10s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 10s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:56:11.246 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 10s ago 5h 79.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:56:11.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 12s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:56:11.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:56:11.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 12s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:56:11.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 10s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:56:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:11.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:11.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:11.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:56:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:56:11.557 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:56:11.829 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[102844]: from='client.74630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[102844]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1496373830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[100622]: from='client.74630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[100622]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:11.830 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1496373830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:12.138 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:56:12.138 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:12.138 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:12.139 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:56:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:11 smithi196 ceph-mon[76349]: from='client.74630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:11 smithi196 ceph-mon[76349]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1496373830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[102844]: from='client.74642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[102844]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[102844]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/797757440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[100622]: from='client.74642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[100622]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[100622]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/797757440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:12 smithi196 ceph-mon[76349]: from='client.74642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:12 smithi196 ceph-mon[76349]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:12 smithi196 ceph-mon[76349]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/797757440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:14.911 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:14 smithi184 ceph-mon[102844]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:14.911 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:14.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:14 smithi184 ceph-mon[100622]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:14.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:15.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:14 smithi196 ceph-mon[76349]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:15.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:16 smithi184 ceph-mon[100622]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:16 smithi184 ceph-mon[102844]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:16 smithi196 ceph-mon[76349]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:18 smithi184 ceph-mon[102844]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:18 smithi184 ceph-mon[100622]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:18 smithi196 ceph-mon[76349]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:20 smithi196 ceph-mon[76349]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[102844]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[100622]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:21.457Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:21.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:21.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:56:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:22 smithi196 ceph-mon[76349]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:22 smithi184 ceph-mon[100622]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:22 smithi184 ceph-mon[102844]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:24 smithi196 ceph-mon[76349]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:24 smithi184 ceph-mon[100622]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:24 smithi184 ceph-mon[102844]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:26 smithi196 ceph-mon[76349]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:26 smithi184 ceph-mon[100622]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:26 smithi184 ceph-mon[102844]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:28 smithi196 ceph-mon[76349]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:28 smithi184 ceph-mon[100622]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:28 smithi184 ceph-mon[102844]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:30 smithi184 ceph-mon[100622]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:30 smithi184 ceph-mon[102844]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:30 smithi196 ceph-mon[76349]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:31.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:31.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:31.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:56:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:32 smithi184 ceph-mon[100622]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:32 smithi184 ceph-mon[102844]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:32 smithi196 ceph-mon[76349]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:35.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:34 smithi184 ceph-mon[100622]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:35.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:34 smithi184 ceph-mon[102844]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:35.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:34 smithi196 ceph-mon[76349]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:36 smithi184 ceph-mon[100622]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:36 smithi184 ceph-mon[102844]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:37.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:36 smithi196 ceph-mon[76349]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:56:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:38 smithi184 ceph-mon[100622]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:38 smithi184 ceph-mon[102844]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:38 smithi196 ceph-mon[76349]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:40.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[102844]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[100622]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:40 smithi196 ceph-mon[76349]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:56:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:41.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:41.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:41.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:56:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:42.433 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:56:42.962 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:56:42.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T00:56:42.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 5h - - 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 502M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 3s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 3s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 3s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 3s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 2s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:56:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 2s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 2s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 2s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 79.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:56:42.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:56:43.276 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:56:43.276 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:56:43.276 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:56:43.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:56:43.277 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:56:43.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:42 smithi184 ceph-mon[100622]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:43.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:42 smithi184 ceph-mon[102844]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:42 smithi196 ceph-mon[76349]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:56:43.546 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:56:43.547 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:56:43.857 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:56:43.857 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:43.857 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:43.857 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[100622]: from='client.64701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[100622]: from='client.74672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[100622]: from='client.64713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1682553636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/265224931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[102844]: from='client.64701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[102844]: from='client.74672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[102844]: from='client.64713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1682553636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/265224931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:43 smithi196 ceph-mon[76349]: from='client.64701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:43 smithi196 ceph-mon[76349]: from='client.74672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:43 smithi196 ceph-mon[76349]: from='client.64713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1682553636' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/265224931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[100622]: from='client.74690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[100622]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[102844]: from='client.74690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[102844]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:44 smithi196 ceph-mon[76349]: from='client.74690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:44 smithi196 ceph-mon[76349]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:47 smithi184 ceph-mon[100622]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:47 smithi184 ceph-mon[102844]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:47 smithi196 ceph-mon[76349]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:49 smithi184 ceph-mon[100622]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:49 smithi184 ceph-mon[102844]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:49 smithi196 ceph-mon[76349]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:51 smithi184 ceph-mon[100622]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:51 smithi184 ceph-mon[102844]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:51 smithi196 ceph-mon[76349]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:51.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:51.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:56:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:56:51.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:56:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:56:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:56:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:53 smithi184 ceph-mon[100622]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:53 smithi184 ceph-mon[102844]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:53 smithi196 ceph-mon[76349]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:55.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:55 smithi196 ceph-mon[76349]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:55 smithi184 ceph-mon[100622]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:55 smithi184 ceph-mon[102844]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:57.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:57 smithi184 ceph-mon[100622]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:57.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:57 smithi184 ceph-mon[102844]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:57 smithi196 ceph-mon[76349]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:56:59 smithi184 ceph-mon[100622]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:56:59 smithi184 ceph-mon[102844]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:56:59 smithi196 ceph-mon[76349]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[100622]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[102844]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:01 smithi196 ceph-mon[76349]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:01.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:01.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:01.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:57:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:03 smithi196 ceph-mon[76349]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:03 smithi184 ceph-mon[100622]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:03.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:03 smithi184 ceph-mon[102844]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:05 smithi196 ceph-mon[76349]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:05 smithi184 ceph-mon[100622]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:05 smithi184 ceph-mon[102844]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:07 smithi196 ceph-mon[76349]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:07 smithi184 ceph-mon[100622]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:07 smithi184 ceph-mon[102844]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:09 smithi196 ceph-mon[76349]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:09 smithi184 ceph-mon[100622]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:09 smithi184 ceph-mon[102844]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:11 smithi184 ceph-mon[100622]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:11 smithi184 ceph-mon[102844]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:11 smithi196 ceph-mon[76349]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:11.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:11.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:11.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:12.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:57:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:13 smithi184 ceph-mon[100622]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:13 smithi184 ceph-mon[102844]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:13 smithi196 ceph-mon[76349]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:14.178 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:57:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 5h 502M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:57:14.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (4h) 15s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 15s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 15s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 15s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 13s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 13s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 13s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 13s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 15s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 15s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:57:14.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:57:15.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:57:15.015 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:57:15.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:57:15.291 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:57:15.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:15 smithi184 ceph-mon[100622]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:15.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1988662254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:15.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:15 smithi184 ceph-mon[102844]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:15.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1988662254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:15 smithi196 ceph-mon[76349]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:15.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1988662254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:15.600 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:57:15.600 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:15.601 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:15.601 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:16.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[100622]: from='client.64737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[100622]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[100622]: from='client.64749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2868944822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[102844]: from='client.64737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[102844]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[102844]: from='client.64749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2868944822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:16.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:16 smithi196 ceph-mon[76349]: from='client.64737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:16 smithi196 ceph-mon[76349]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:16 smithi196 ceph-mon[76349]: from='client.64749 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:16.551 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2868944822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:17 smithi184 ceph-mon[102844]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:17 smithi184 ceph-mon[102844]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:17 smithi184 ceph-mon[100622]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:17 smithi184 ceph-mon[100622]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:17 smithi196 ceph-mon[76349]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:17 smithi196 ceph-mon[76349]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:19.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:19 smithi184 ceph-mon[100622]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:19.517 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:19 smithi184 ceph-mon[102844]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:19 smithi196 ceph-mon[76349]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[102844]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:20 smithi184 ceph-mon[100622]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:20 smithi196 ceph-mon[76349]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:21.461Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:21.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:21.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:22 smithi196 ceph-mon[76349]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:22 smithi184 ceph-mon[100622]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:22 smithi184 ceph-mon[102844]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:24 smithi196 ceph-mon[76349]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:24 smithi184 ceph-mon[100622]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:24 smithi184 ceph-mon[102844]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:26 smithi196 ceph-mon[76349]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:26 smithi184 ceph-mon[100622]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:26 smithi184 ceph-mon[102844]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:28 smithi196 ceph-mon[76349]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:28 smithi184 ceph-mon[102844]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:28 smithi184 ceph-mon[100622]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:30 smithi196 ceph-mon[76349]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:30 smithi184 ceph-mon[100622]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:30 smithi184 ceph-mon[102844]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:31.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:31.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:31.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:31 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:31 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:31 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:57:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:32 smithi184 ceph-mon[100622]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:32 smithi184 ceph-mon[102844]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:32 smithi196 ceph-mon[76349]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:34 smithi196 ceph-mon[76349]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:34 smithi184 ceph-mon[100622]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:34 smithi184 ceph-mon[102844]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:36 smithi196 ceph-mon[76349]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:37.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:36 smithi184 ceph-mon[100622]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:37.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:36 smithi184 ceph-mon[102844]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:38 smithi184 ceph-mon[100622]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:38 smithi184 ceph-mon[102844]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:38 smithi196 ceph-mon[76349]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:40 smithi196 ceph-mon[76349]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[100622]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:40 smithi184 ceph-mon[102844]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:41.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:41.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:41.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:57:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:42 smithi184 ceph-mon[100622]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:42 smithi184 ceph-mon[102844]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:42 smithi196 ceph-mon[76349]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:44 smithi184 ceph-mon[100622]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:44 smithi184 ceph-mon[102844]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:44 smithi196 ceph-mon[76349]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:45.897 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:57:46.428 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 5h 502M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 7s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 7s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 7s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 5s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 5s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 5s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 5s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:57:46.429 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:57:46.430 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:57:46.430 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:57:46.737 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:57:46.737 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:57:46.738 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:57:46.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[100622]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:46.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[100622]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:46.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[100622]: from='client.74744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:46.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3956626133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:47.008 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:57:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:57:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:57:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:57:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:57:47.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:57:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[102844]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[102844]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[102844]: from='client.74744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3956626133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:47.316 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:57:47.316 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:47.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:47.316 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:47.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:46 smithi196 ceph-mon[76349]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:46 smithi196 ceph-mon[76349]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:46 smithi196 ceph-mon[76349]: from='client.74744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3956626133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[100622]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[100622]: from='client.64797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4003234301' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[102844]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[102844]: from='client.64797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4003234301' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:47 smithi196 ceph-mon[76349]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:47 smithi196 ceph-mon[76349]: from='client.64797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4003234301' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:48 smithi184 ceph-mon[100622]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:48 smithi184 ceph-mon[102844]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:48 smithi196 ceph-mon[76349]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:50 smithi184 ceph-mon[100622]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:50 smithi184 ceph-mon[102844]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:50 smithi196 ceph-mon[76349]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:51.463Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:51.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:57:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:57:51.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:57:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:57:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T00:57:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:53 smithi184 ceph-mon[100622]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:53 smithi184 ceph-mon[102844]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:53 smithi196 ceph-mon[76349]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:55 smithi184 ceph-mon[100622]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:55 smithi184 ceph-mon[102844]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:55 smithi196 ceph-mon[76349]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:57 smithi196 ceph-mon[76349]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:57 smithi184 ceph-mon[100622]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:57 smithi184 ceph-mon[102844]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:57:59 smithi184 ceph-mon[100622]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:57:59 smithi184 ceph-mon[102844]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:57:59 smithi196 ceph-mon[76349]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:01 smithi184 ceph-mon[100622]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:01 smithi184 ceph-mon[102844]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:01 smithi196 ceph-mon[76349]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:01.464Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:01.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:01.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:02] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:03 smithi196 ceph-mon[76349]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:03 smithi184 ceph-mon[100622]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:03 smithi184 ceph-mon[102844]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:05 smithi196 ceph-mon[76349]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:05 smithi184 ceph-mon[100622]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:05 smithi184 ceph-mon[102844]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:07 smithi196 ceph-mon[76349]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:07 smithi184 ceph-mon[100622]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:07 smithi184 ceph-mon[102844]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:09 smithi196 ceph-mon[76349]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:09 smithi184 ceph-mon[100622]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:09.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:09 smithi184 ceph-mon[102844]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:11 smithi184 ceph-mon[100622]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:11 smithi184 ceph-mon[102844]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:11 smithi196 ceph-mon[76349]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:11.465Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:11.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:11.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:12] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:58:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:13 smithi196 ceph-mon[76349]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:13 smithi184 ceph-mon[100622]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:13 smithi184 ceph-mon[102844]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:14.135 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 00:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T00:58:13.812327745Z level=info msg="Completed cleanup jobs" duration=12.507757ms 2024-09-21T00:58:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:15 smithi196 ceph-mon[76349]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:15 smithi184 ceph-mon[100622]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:15 smithi184 ceph-mon[102844]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:17 smithi184 ceph-mon[100622]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:17 smithi184 ceph-mon[102844]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:17.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:17 smithi196 ceph-mon[76349]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:17.612 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:58:18.138 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 18s ago 5h 502M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 18s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:58:18.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (4h) 18s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 18s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 18s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 17s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 17s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:58:18.140 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:18.450 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:58:18.451 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:58:18.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:58:18.724 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:58:18.724 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:58:18.724 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:58:18.724 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:58:18.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:58:19.047 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:58:19.047 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:19.047 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:19.047 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:19.047 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: from='client.64809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: from='client.64815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4166126686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1604440577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: from='client.64809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: from='client.64815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4166126686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1604440577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: from='client.64809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: from='client.64815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4166126686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1604440577' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:20.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[100622]: from='client.64833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:20.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[102844]: from='client.64833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:20 smithi196 ceph-mon[76349]: from='client.64833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:21 smithi184 ceph-mon[100622]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:21 smithi184 ceph-mon[102844]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:21 smithi196 ceph-mon[76349]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:21.465Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:21.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:21.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:23 smithi184 ceph-mon[100622]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:23 smithi184 ceph-mon[102844]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:23 smithi196 ceph-mon[76349]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:25 smithi184 ceph-mon[100622]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:25 smithi184 ceph-mon[102844]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:25 smithi196 ceph-mon[76349]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:27 smithi184 ceph-mon[100622]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:27 smithi184 ceph-mon[102844]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:27 smithi196 ceph-mon[76349]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:29 smithi184 ceph-mon[100622]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:29 smithi184 ceph-mon[102844]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:29 smithi196 ceph-mon[76349]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:31 smithi184 ceph-mon[100622]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:31 smithi184 ceph-mon[102844]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:31 smithi196 ceph-mon[76349]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:31.466Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:31.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:31.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:32] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:58:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:33 smithi184 ceph-mon[100622]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:33 smithi184 ceph-mon[102844]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:33 smithi196 ceph-mon[76349]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:34 smithi184 ceph-mon[100622]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:34 smithi184 ceph-mon[102844]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:34 smithi196 ceph-mon[76349]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:36 smithi184 ceph-mon[100622]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:36 smithi184 ceph-mon[102844]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:36 smithi196 ceph-mon[76349]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:38.819 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:38 smithi196 ceph-mon[76349]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:38 smithi184 ceph-mon[100622]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:38 smithi184 ceph-mon[102844]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:40.829 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[100622]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:40.829 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:40.829 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:40.829 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:40.829 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[102844]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:40.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:40.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:40.830 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:41.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:40 smithi196 ceph-mon[76349]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:41.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:41.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:41.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:41.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:41.467Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:41.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:41.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:41.757 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:41.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:42.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:41 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:42.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:42] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T00:58:42.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:42.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:58:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:42 smithi184 ceph-mon[100622]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:42 smithi184 ceph-mon[102844]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:42 smithi196 ceph-mon[76349]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:44 smithi184 ceph-mon[102844]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:44 smithi184 ceph-mon[100622]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:44 smithi196 ceph-mon[76349]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:46 smithi184 ceph-mon[100622]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:46 smithi184 ceph-mon[102844]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:46 smithi196 ceph-mon[76349]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:48 smithi184 ceph-mon[100622]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:48 smithi184 ceph-mon[102844]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:48 smithi196 ceph-mon[76349]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:49.345 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:58:49.875 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:58:49.875 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 5h - - 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 5h - - 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 10s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 10s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 10s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 10s ago 5h 503M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 5h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 10s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 10s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 10s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:58:49.876 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 10s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (4h) 10s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 10s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 8s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 8s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 8s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 8s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 77.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:58:49.877 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:58:50.188 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:58:50.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:58:50.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:58:50.190 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:58:50.455 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:58:50.456 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:58:50.763 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:58:50.763 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:50.763 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:50.763 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:50.763 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: from='client.74810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: from='client.64857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2946493234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1129667232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: from='client.74810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: from='client.64857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2946493234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1129667232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: from='client.74810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: from='client.64857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2946493234' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1129667232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:51.778 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:51.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:51.778 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:51.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:58:51.778 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:58:51.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:58:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:58:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T00:58:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-mon[100622]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:52.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-mon[102844]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:52.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:51 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:52.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:51 smithi196 ceph-mon[76349]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:52.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:51 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:52 smithi184 ceph-mon[102844]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:52 smithi184 ceph-mon[100622]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:52 smithi196 ceph-mon[76349]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:55.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:54 smithi184 ceph-mon[102844]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:55.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:54 smithi184 ceph-mon[100622]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:54 smithi196 ceph-mon[76349]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:56 smithi184 ceph-mon[100622]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:56 smithi184 ceph-mon[102844]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:56 smithi196 ceph-mon[76349]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:58 smithi184 ceph-mon[100622]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:58 smithi184 ceph-mon[102844]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:58 smithi196 ceph-mon[76349]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:00 smithi196 ceph-mon[76349]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[100622]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[102844]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:01.469Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:01.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:01.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:59:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:02 smithi184 ceph-mon[100622]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:02 smithi184 ceph-mon[102844]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:02 smithi196 ceph-mon[76349]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:04 smithi184 ceph-mon[100622]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:04 smithi184 ceph-mon[102844]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:04 smithi196 ceph-mon[76349]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:06 smithi184 ceph-mon[100622]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:06 smithi184 ceph-mon[102844]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:06 smithi196 ceph-mon[76349]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:08 smithi184 ceph-mon[100622]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:08 smithi184 ceph-mon[102844]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:08 smithi196 ceph-mon[76349]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:11.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:10 smithi196 ceph-mon[76349]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:10 smithi184 ceph-mon[100622]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:10 smithi184 ceph-mon[102844]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:11.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:11.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:11.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:12.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:12.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:59:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:12 smithi184 ceph-mon[100622]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:12 smithi184 ceph-mon[102844]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:13.355 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:12 smithi196 ceph-mon[76349]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:14 smithi184 ceph-mon[100622]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:14 smithi184 ceph-mon[102844]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:14 smithi196 ceph-mon[76349]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:16 smithi184 ceph-mon[100622]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:16 smithi184 ceph-mon[102844]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:16 smithi196 ceph-mon[76349]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:18 smithi184 ceph-mon[102844]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:18 smithi184 ceph-mon[100622]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:18 smithi196 ceph-mon[76349]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[102844]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[100622]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.068 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:59:21.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:21 smithi196 ceph-mon[76349]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:21.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.252 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 5h - - 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 20s ago 5h - - 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 1s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 20s ago 5h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 1s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 20s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 1s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:59:21.600 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 1s ago 5h 504M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 20s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 1s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 1s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 20s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 1s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (4h) 1s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 20s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 20s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 20s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 20s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 20s ago 5h 77.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:59:21.601 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 1s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:59:21.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 20s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:59:21.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 1s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:59:21.602 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 20s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:21.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:21.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:21.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:21.918 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:21.919 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:59:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:59:21.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:59:21.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:59:22.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='client.64881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3296982020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T00:59:22.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='client.64881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3296982020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:22.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:59:22.201 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:59:22.202 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='client.64881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3296982020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:22.526 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:59:22.527 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:22.527 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:22.527 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:22.527 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[102844]: from='client.74852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[102844]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[102844]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2297391081' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[100622]: from='client.74852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[100622]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[100622]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2297391081' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:23 smithi196 ceph-mon[76349]: from='client.74852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:23 smithi196 ceph-mon[76349]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:23 smithi196 ceph-mon[76349]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2297391081' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:24.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:24 smithi184 ceph-mon[100622]: from='client.74870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:24.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:24 smithi184 ceph-mon[102844]: from='client.74870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:24 smithi196 ceph-mon[76349]: from='client.74870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:25 smithi184 ceph-mon[100622]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:25 smithi184 ceph-mon[102844]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:25 smithi196 ceph-mon[76349]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:27 smithi184 ceph-mon[100622]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:27 smithi184 ceph-mon[102844]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:27 smithi196 ceph-mon[76349]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:29 smithi184 ceph-mon[100622]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:29 smithi184 ceph-mon[102844]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:29 smithi196 ceph-mon[76349]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:31 smithi184 ceph-mon[100622]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:31 smithi184 ceph-mon[102844]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:31 smithi196 ceph-mon[76349]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:31.471Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:31.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:31.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:59:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:33 smithi196 ceph-mon[76349]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:33 smithi184 ceph-mon[100622]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:33 smithi184 ceph-mon[102844]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:35 smithi196 ceph-mon[76349]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:35 smithi184 ceph-mon[100622]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:35 smithi184 ceph-mon[102844]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:37 smithi196 ceph-mon[76349]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:37 smithi184 ceph-mon[102844]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:37 smithi184 ceph-mon[100622]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:39 smithi196 ceph-mon[76349]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:39 smithi184 ceph-mon[100622]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:39 smithi184 ceph-mon[102844]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:41.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[100622]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:41.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[102844]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:41.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.531 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:41 smithi196 ceph-mon[76349]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:41.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:41.471Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:41.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:41.704 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:41.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T00:59:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T00:59:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:43 smithi196 ceph-mon[76349]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:43 smithi184 ceph-mon[100622]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:43 smithi184 ceph-mon[102844]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:45 smithi184 ceph-mon[100622]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:45 smithi184 ceph-mon[102844]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:45 smithi196 ceph-mon[76349]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:47 smithi184 ceph-mon[100622]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:47 smithi184 ceph-mon[102844]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:47 smithi196 ceph-mon[76349]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:49 smithi184 ceph-mon[100622]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:49.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:49 smithi184 ceph-mon[102844]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:49 smithi196 ceph-mon[76349]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:51.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:51 smithi184 ceph-mon[100622]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:51.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:51 smithi184 ceph-mon[102844]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:51 smithi196 ceph-mon[76349]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:51.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:51.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T00:59:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 00:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T00:59:51.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T00:59:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 00:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:00:59:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T00:59:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:52.843 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 5h - - 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 5h - - 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 13s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 11s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 13s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 11s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 13s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T00:59:53.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 13s ago 5h 507M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 11s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 13s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 13s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 11s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 13s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 13s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 13s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 13s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 11s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 11s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 11s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 11s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 11s ago 5h 78.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T00:59:53.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 13s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T00:59:53.374 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T00:59:53.374 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 13s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T00:59:53.374 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 11s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T00:59:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:53 smithi184 ceph-mon[100622]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:53 smithi184 ceph-mon[102844]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:53 smithi196 ceph-mon[76349]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T00:59:53.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T00:59:53.686 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:59:53.954 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T00:59:53.954 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:59:53.954 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T00:59:53.954 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:59:53.954 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T00:59:53.955 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T00:59:54.267 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:59:54.267 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:54.267 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:54.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[100622]: from='client.64917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[100622]: from='client.74888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1810215782' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:54.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[102844]: from='client.64917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[102844]: from='client.74888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1810215782' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:54 smithi196 ceph-mon[76349]: from='client.64917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:54 smithi196 ceph-mon[76349]: from='client.74888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1810215782' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[100622]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[100622]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[100622]: from='client.64941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3760606588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[102844]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[102844]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[102844]: from='client.64941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3760606588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:55 smithi196 ceph-mon[76349]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:55 smithi196 ceph-mon[76349]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:55 smithi196 ceph-mon[76349]: from='client.64941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3760606588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:57 smithi184 ceph-mon[100622]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:57 smithi184 ceph-mon[102844]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:57 smithi196 ceph-mon[76349]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:58 smithi184 ceph-mon[100622]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:58 smithi184 ceph-mon[102844]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:58 smithi196 ceph-mon[76349]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 00:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 00:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 00:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:00.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:00 smithi184 ceph-mon[102844]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.287 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:00.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T00:59:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:00.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:00 smithi184 ceph-mon[100622]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.287 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:00 smithi196 ceph-mon[76349]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:01.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.473 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:01.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.473 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:01.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:01.483 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:01.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:01.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:01.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:02.277 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:00:02.355 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T01:00:02.035Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726869600138 maxt=1726876800000 ulid=01J891FJJM4EZE63523AXR48J7 duration=31.46818ms 2024-09-21T01:00:02.355 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T01:00:02.037Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=1.966454ms 2024-09-21T01:00:02.355 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T01:00:02.038Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 mint=1726876800000 2024-09-21T01:00:02.356 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T01:00:02.100Z caller=head.go:1241 level=info component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=62.513652ms 2024-09-21T01:00:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[102844]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[100622]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-mon[76349]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:03.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:03.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:04 smithi184 ceph-mon[100622]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:04 smithi184 ceph-mon[102844]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:04 smithi196 ceph-mon[76349]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:06 smithi184 ceph-mon[100622]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:06 smithi184 ceph-mon[102844]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:06 smithi196 ceph-mon[76349]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:08 smithi184 ceph-mon[100622]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:08 smithi184 ceph-mon[102844]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:08 smithi196 ceph-mon[76349]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:11.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:10 smithi196 ceph-mon[76349]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:10 smithi184 ceph-mon[100622]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:10 smithi184 ceph-mon[102844]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:11.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:11.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:11.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:00:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:12 smithi184 ceph-mon[100622]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:12 smithi184 ceph-mon[102844]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:12 smithi196 ceph-mon[76349]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:14 smithi184 ceph-mon[102844]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:14 smithi184 ceph-mon[100622]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:14 smithi196 ceph-mon[76349]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:16 smithi184 ceph-mon[100622]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:16 smithi184 ceph-mon[102844]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:16 smithi196 ceph-mon[76349]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:18 smithi184 ceph-mon[100622]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:18 smithi184 ceph-mon[102844]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:18 smithi196 ceph-mon[76349]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[102844]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[100622]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:20 smithi196 ceph-mon[76349]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:21.474Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:21.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:21.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:21 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:21 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:21 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:00:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:23 smithi196 ceph-mon[76349]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.233 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[100622]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:23.233 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[102844]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:23.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:24.562 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 5h - - 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 4s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 3s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 4s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:00:25.130 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 4s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 4s ago 5h 504M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 4s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 4s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 3s ago 5h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 4s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 4s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 3s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 3s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 3s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:00:25.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 3s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:00:25.132 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 5h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:00:25.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 4s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:00:25.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:00:25.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 4s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:00:25.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:00:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:25 smithi184 ceph-mon[100622]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:25 smithi184 ceph-mon[102844]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:25 smithi196 ceph-mon[76349]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:25.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:00:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:00:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:00:25.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:00:25.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:00:25.715 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:00:25.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:00:26.024 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:26.024 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:26.024 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[100622]: from='client.74918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[100622]: from='client.74924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[100622]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1123715553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3844959028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[102844]: from='client.74918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[102844]: from='client.74924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[102844]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1123715553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3844959028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:26 smithi196 ceph-mon[76349]: from='client.74918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:26 smithi196 ceph-mon[76349]: from='client.74924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:26 smithi196 ceph-mon[76349]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1123715553' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3844959028' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:27 smithi196 ceph-mon[76349]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:27 smithi196 ceph-mon[76349]: from='client.74942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:27 smithi184 ceph-mon[100622]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:27 smithi184 ceph-mon[100622]: from='client.74942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:27 smithi184 ceph-mon[102844]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:27 smithi184 ceph-mon[102844]: from='client.74942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:29 smithi196 ceph-mon[76349]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:29 smithi184 ceph-mon[100622]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:29 smithi184 ceph-mon[102844]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:31 smithi196 ceph-mon[76349]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:31.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:31 smithi184 ceph-mon[100622]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:31.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:31 smithi184 ceph-mon[102844]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:31.475Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:31.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:31.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:00:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:33 smithi196 ceph-mon[76349]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:00:33.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:33 smithi184 ceph-mon[100622]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:00:33.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:33 smithi184 ceph-mon[102844]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:00:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:35 smithi196 ceph-mon[76349]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:35 smithi184 ceph-mon[100622]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:35 smithi184 ceph-mon[102844]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:37 smithi196 ceph-mon[76349]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:37 smithi184 ceph-mon[100622]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:37 smithi184 ceph-mon[102844]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:39 smithi196 ceph-mon[76349]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:39 smithi184 ceph-mon[100622]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:39 smithi184 ceph-mon[102844]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:41 smithi196 ceph-mon[76349]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[100622]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[102844]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:41.476Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:41.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:41.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:00:42.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[100622]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[102844]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:43 smithi196 ceph-mon[76349]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:43.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:43.420 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:00:44.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:44.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:44.546 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:45.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:45 smithi184 ceph-mon[100622]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:45.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:45 smithi184 ceph-mon[102844]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:45 smithi196 ceph-mon[76349]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:47 smithi184 ceph-mon[100622]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:47 smithi184 ceph-mon[102844]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:47 smithi196 ceph-mon[76349]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:49 smithi184 ceph-mon[100622]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:49 smithi184 ceph-mon[102844]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:49 smithi196 ceph-mon[76349]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:51.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:51 smithi184 ceph-mon[100622]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:51.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:51 smithi184 ceph-mon[102844]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:51 smithi196 ceph-mon[76349]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:51.477Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:51.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:00:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:00:51.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:00:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:00:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:00:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:53 smithi184 ceph-mon[100622]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:53 smithi184 ceph-mon[102844]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:53 smithi196 ceph-mon[76349]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:55 smithi184 ceph-mon[100622]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:55 smithi184 ceph-mon[102844]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:55 smithi196 ceph-mon[76349]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:56.327 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:00:56.853 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:00:56.853 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 5h - - 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 5h - - 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 16s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 14s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 16s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 14s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 16s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 16s ago 5h 505M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 14s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 16s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 16s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 14s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:00:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 16s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 16s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 16s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 16s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (4h) 14s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 14s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 14s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 14s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 14s ago 5h 77.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 16s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 16s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:00:56.855 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 14s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:00:57.163 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:00:57.163 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:00:57.163 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:00:57.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:57.163 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:00:57.164 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:00:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:00:57.432 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:00:57.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:57 smithi184 ceph-mon[100622]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:57.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2356176302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:57.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:57 smithi184 ceph-mon[102844]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:57.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2356176302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:57 smithi196 ceph-mon[76349]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2356176302' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:57.740 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:57.740 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:57.741 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:57.741 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:57.741 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[102844]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[102844]: from='client.74960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[102844]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/109226993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[100622]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[100622]: from='client.74960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[100622]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/109226993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:58 smithi196 ceph-mon[76349]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:58 smithi196 ceph-mon[76349]: from='client.74960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:58 smithi196 ceph-mon[76349]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/109226993' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[102844]: from='client.65013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[102844]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[100622]: from='client.65013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[100622]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:59 smithi196 ceph-mon[76349]: from='client.65013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:59 smithi196 ceph-mon[76349]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:00 smithi184 ceph-mon[102844]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:00 smithi184 ceph-mon[100622]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:00 smithi196 ceph-mon[76349]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:01.477Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:01.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:01.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:01.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:02.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:02.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:02.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:02.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:01 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:01:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[102844]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[100622]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:02 smithi196 ceph-mon[76349]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:03.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:03.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:04.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:04 smithi196 ceph-mon[76349]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:04 smithi184 ceph-mon[100622]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:04 smithi184 ceph-mon[102844]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:06 smithi184 ceph-mon[100622]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:06 smithi184 ceph-mon[102844]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:06 smithi196 ceph-mon[76349]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:09.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:08 smithi184 ceph-mon[100622]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:09.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:08 smithi184 ceph-mon[102844]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:08 smithi196 ceph-mon[76349]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:10 smithi184 ceph-mon[100622]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:10 smithi184 ceph-mon[102844]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:10 smithi196 ceph-mon[76349]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:11.765 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:11.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:11.765 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:11.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:11.765 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:11.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:12.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:11 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:12.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:11 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:12.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:11 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:01:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:12 smithi184 ceph-mon[100622]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:12 smithi184 ceph-mon[102844]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:13.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:12 smithi196 ceph-mon[76349]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:14 smithi184 ceph-mon[102844]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:14 smithi184 ceph-mon[100622]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:14 smithi196 ceph-mon[76349]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:16 smithi184 ceph-mon[100622]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:16 smithi184 ceph-mon[102844]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:16 smithi196 ceph-mon[76349]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:18 smithi184 ceph-mon[100622]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:18 smithi184 ceph-mon[102844]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:18 smithi196 ceph-mon[76349]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:20 smithi184 ceph-mon[100622]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:20 smithi184 ceph-mon[102844]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:20 smithi196 ceph-mon[76349]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:21.785 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:21.479Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:21.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:21.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:21.925 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:21.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:22.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T01:01:22.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[102844]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[100622]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:22 smithi196 ceph-mon[76349]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:25.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:24 smithi184 ceph-mon[100622]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:25.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:24 smithi184 ceph-mon[102844]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:25.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:24 smithi196 ceph-mon[76349]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:26 smithi196 ceph-mon[76349]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:26 smithi184 ceph-mon[100622]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:26 smithi184 ceph-mon[102844]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:28.044 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 6s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 5h 506M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:01:28.592 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 6s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 7s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 7s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 7s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (4h) 6s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 6s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 6s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 5h 63.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 7s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:01:28.593 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 6s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:01:28.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:01:28.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:01:28.913 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:01:28.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:01:28.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:01:28.913 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:01:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:28 smithi196 ceph-mon[76349]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:28 smithi196 ceph-mon[76349]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:01:29.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:01:29.182 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:01:29.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:28 smithi184 ceph-mon[100622]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:29.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:28 smithi184 ceph-mon[100622]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:29.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:28 smithi184 ceph-mon[102844]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:29.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:28 smithi184 ceph-mon[102844]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:29.490 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:01:29.490 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:29.490 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:29.490 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:29.490 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='client.65031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1199196771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1666556143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='client.65031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1199196771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1666556143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='client.65031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1199196771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1666556143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:31.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:30 smithi184 ceph-mon[100622]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:31.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:30 smithi184 ceph-mon[102844]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:30 smithi196 ceph-mon[76349]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:31.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:31.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:31.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:01:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:32 smithi184 ceph-mon[102844]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:32 smithi184 ceph-mon[100622]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:32 smithi196 ceph-mon[76349]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:34 smithi184 ceph-mon[100622]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:34 smithi184 ceph-mon[102844]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:34 smithi196 ceph-mon[76349]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:36 smithi184 ceph-mon[100622]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:36 smithi184 ceph-mon[102844]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:36 smithi196 ceph-mon[76349]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:38 smithi184 ceph-mon[100622]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:38 smithi184 ceph-mon[102844]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:38 smithi196 ceph-mon[76349]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:41.224 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:40 smithi184 ceph-mon[100622]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:41.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:40 smithi184 ceph-mon[102844]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:40 smithi196 ceph-mon[76349]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:41.481Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:41.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:41.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:42.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:01:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[100622]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[102844]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:42 smithi196 ceph-mon[76349]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:01:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:44 smithi196 ceph-mon[76349]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:44 smithi184 ceph-mon[100622]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:44 smithi184 ceph-mon[102844]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:46 smithi196 ceph-mon[76349]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:47.144 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:46 smithi184 ceph-mon[100622]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:47.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:46 smithi184 ceph-mon[102844]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:01:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:48 smithi184 ceph-mon[100622]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:48 smithi184 ceph-mon[102844]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:48 smithi196 ceph-mon[76349]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:51.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:50 smithi184 ceph-mon[100622]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:51.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:50 smithi184 ceph-mon[102844]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:50 smithi196 ceph-mon[76349]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:51.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:51.481Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:51.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:01:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:01:51.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:01:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:01:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:01:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:52 smithi184 ceph-mon[100622]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:52 smithi184 ceph-mon[102844]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:52 smithi196 ceph-mon[76349]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:54 smithi184 ceph-mon[100622]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:54 smithi184 ceph-mon[102844]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:54 smithi196 ceph-mon[76349]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:56 smithi184 ceph-mon[100622]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:56 smithi184 ceph-mon[102844]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:56 smithi196 ceph-mon[76349]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:58 smithi184 ceph-mon[100622]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:58 smithi184 ceph-mon[102844]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:58 smithi196 ceph-mon[76349]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:59.790 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:02:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 19s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 92.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:02:00.315 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 5h 506M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 5h 350M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 19s ago 5h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 17s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 17s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 64.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:02:00.316 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 19s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:02:00.317 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:02:00.317 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 19s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:02:00.317 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:02:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:02:00.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:02:00.628 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:02:00.894 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:02:00.895 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:02:00.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:02:01.212 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:02:01.213 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:01.213 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:01.213 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:01.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[100622]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:01.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[100622]: from='client.75026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[100622]: from='client.65067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1586123705' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:01.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[102844]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:01.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[102844]: from='client.75026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[102844]: from='client.65067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1586123705' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:00 smithi196 ceph-mon[76349]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:00 smithi196 ceph-mon[76349]: from='client.75026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:00 smithi196 ceph-mon[76349]: from='client.65067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:01.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1586123705' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:01.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:01.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:01.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:02:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2002752024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2002752024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2002752024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[100622]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:03.310 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[102844]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:03.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.311 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:03 smithi196 ceph-mon[76349]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:05.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:05 smithi196 ceph-mon[76349]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:05 smithi184 ceph-mon[100622]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:05 smithi184 ceph-mon[102844]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:07.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:07 smithi196 ceph-mon[76349]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:07.419 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:07 smithi184 ceph-mon[100622]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:07.419 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:07 smithi184 ceph-mon[102844]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:09 smithi196 ceph-mon[76349]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:09 smithi184 ceph-mon[100622]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:09 smithi184 ceph-mon[102844]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:11 smithi196 ceph-mon[76349]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:11.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:11 smithi184 ceph-mon[100622]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:11.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:11 smithi184 ceph-mon[102844]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:11.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:11.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:11.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:02:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:13 smithi196 ceph-mon[76349]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:13 smithi184 ceph-mon[100622]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:13 smithi184 ceph-mon[102844]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:14.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:14.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:15 smithi196 ceph-mon[76349]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:15 smithi184 ceph-mon[100622]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:15 smithi184 ceph-mon[102844]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:17 smithi196 ceph-mon[76349]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:17 smithi184 ceph-mon[100622]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:17 smithi184 ceph-mon[102844]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:19 smithi196 ceph-mon[76349]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:19 smithi184 ceph-mon[100622]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:19 smithi184 ceph-mon[102844]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:21 smithi196 ceph-mon[76349]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:21.483 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:21 smithi184 ceph-mon[100622]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:21.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:21 smithi184 ceph-mon[102844]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:21.483Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:21.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:21.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:02:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[102844]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[100622]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:23 smithi196 ceph-mon[76349]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:25.457 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:25 smithi196 ceph-mon[76349]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:25 smithi184 ceph-mon[100622]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:25 smithi184 ceph-mon[102844]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:27.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:27 smithi184 ceph-mon[100622]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:27.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:27 smithi184 ceph-mon[102844]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:27.519 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:27 smithi196 ceph-mon[76349]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:29 smithi184 ceph-mon[100622]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:29 smithi184 ceph-mon[102844]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:29 smithi196 ceph-mon[76349]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:31.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:31 smithi184 ceph-mon[100622]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:31.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:31 smithi184 ceph-mon[102844]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:31.509 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:02:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:31 smithi196 ceph-mon[76349]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:31.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:31.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:31.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:32.040 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 5h - - 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 10s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 92.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 10s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 10s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 10s ago 5h 507M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 10s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 10s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:02:32.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 10s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 10s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 10s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (4h) 9s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 9s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 5h 65.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 10s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:02:32.042 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 10s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:02:32.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 9s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:02:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:32] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:02:32.356 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:02:32.356 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:02:32.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:02:32.357 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:02:32.623 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:02:32.624 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:02:32.933 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:02:32.933 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:32.934 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:32.934 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:32.934 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.65097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1662969217' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:33.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1573929564' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.65097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1662969217' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:33.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1573929564' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.65097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1662969217' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1573929564' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:34.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:34 smithi184 ceph-mon[100622]: from='client.75086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:34.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:34 smithi184 ceph-mon[102844]: from='client.75086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:34 smithi196 ceph-mon[76349]: from='client.75086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:35 smithi184 ceph-mon[100622]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:35 smithi184 ceph-mon[102844]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:35 smithi196 ceph-mon[76349]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:37 smithi184 ceph-mon[100622]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:37 smithi184 ceph-mon[102844]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:37 smithi196 ceph-mon[76349]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:39 smithi184 ceph-mon[100622]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:39 smithi184 ceph-mon[102844]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:39 smithi196 ceph-mon[76349]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:40 smithi184 ceph-mon[100622]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:40 smithi184 ceph-mon[102844]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:40 smithi196 ceph-mon[76349]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:41.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:41.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:41.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:41.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:41.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:41.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:42.290 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:42] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:02:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[100622]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:42 smithi184 ceph-mon[102844]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:42 smithi196 ceph-mon[76349]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:44 smithi184 ceph-mon[102844]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:44 smithi184 ceph-mon[100622]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:44 smithi196 ceph-mon[76349]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:47.225 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:46 smithi196 ceph-mon[76349]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:46 smithi184 ceph-mon[100622]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:46 smithi184 ceph-mon[102844]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:49.187 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:48 smithi184 ceph-mon[100622]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:49.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:48 smithi184 ceph-mon[102844]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:48 smithi196 ceph-mon[76349]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:51.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:50 smithi184 ceph-mon[100622]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:51.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:50 smithi184 ceph-mon[102844]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:50 smithi196 ceph-mon[76349]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:51.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:51.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:51.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:02:51.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:02:51.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:02:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:02:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:02:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:52 smithi184 ceph-mon[100622]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:52 smithi184 ceph-mon[102844]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:52 smithi196 ceph-mon[76349]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:54 smithi184 ceph-mon[100622]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:54 smithi184 ceph-mon[102844]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:54 smithi196 ceph-mon[76349]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:56 smithi184 ceph-mon[100622]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:56 smithi184 ceph-mon[102844]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:56 smithi196 ceph-mon[76349]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:58 smithi184 ceph-mon[100622]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:58 smithi184 ceph-mon[102844]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:58 smithi196 ceph-mon[76349]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:01.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:00 smithi184 ceph-mon[100622]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:01.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:00 smithi184 ceph-mon[102844]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:00 smithi196 ceph-mon[76349]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:01.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:01.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:01.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:01.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:01.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:03:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[100622]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:02 smithi184 ceph-mon[102844]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:02 smithi196 ceph-mon[76349]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:03.242 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 5h - - 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 0s ago 5h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 0s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 5h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 5h 507M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:03:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 0s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 0s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 0s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 0s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 0s ago 5h 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 2s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:03:03.798 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 2s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:03:03.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 0s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:03:04.120 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:03:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:03:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:03:04.121 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:03:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:03:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:03:04.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:03:04.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.401 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:04.724 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:03:04.725 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:04.725 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:04.725 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: from='client.65133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: from='client.65139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: from='client.65145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4009880168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/153419575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: from='client.65133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: from='client.65139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: from='client.65145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4009880168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/153419575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: from='client.65133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: from='client.65139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: from='client.65145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4009880168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/153419575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:06 smithi196 ceph-mon[76349]: from='client.75122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:06 smithi184 ceph-mon[100622]: from='client.75122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:06 smithi184 ceph-mon[102844]: from='client.75122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:07 smithi196 ceph-mon[76349]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:07 smithi184 ceph-mon[100622]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:07 smithi184 ceph-mon[102844]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:09.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:09 smithi184 ceph-mon[100622]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:09.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:09 smithi184 ceph-mon[102844]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:09 smithi196 ceph-mon[76349]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:11.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:11 smithi184 ceph-mon[100622]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:11.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:11 smithi184 ceph-mon[102844]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:11 smithi196 ceph-mon[76349]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:11.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:11.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:11.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:03:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:13 smithi196 ceph-mon[76349]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:13 smithi184 ceph-mon[100622]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:13 smithi184 ceph-mon[102844]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:14.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:15 smithi196 ceph-mon[76349]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:15 smithi184 ceph-mon[100622]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:15 smithi184 ceph-mon[102844]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:17 smithi196 ceph-mon[76349]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:17 smithi184 ceph-mon[100622]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:17 smithi184 ceph-mon[102844]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:19 smithi196 ceph-mon[76349]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:19 smithi184 ceph-mon[100622]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:19 smithi184 ceph-mon[102844]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:21 smithi196 ceph-mon[76349]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:21.487 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:21 smithi184 ceph-mon[100622]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:21.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:21 smithi184 ceph-mon[102844]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:21.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:21.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:21.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:03:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[100622]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:22 smithi184 ceph-mon[102844]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:22 smithi196 ceph-mon[76349]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:25 smithi184 ceph-mon[100622]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:25 smithi184 ceph-mon[102844]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:25 smithi196 ceph-mon[76349]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:26 smithi184 ceph-mon[100622]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:26 smithi184 ceph-mon[102844]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:26 smithi196 ceph-mon[76349]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:28 smithi196 ceph-mon[76349]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:29.070 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:28 smithi184 ceph-mon[100622]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:29.070 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:28 smithi184 ceph-mon[102844]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:30 smithi196 ceph-mon[76349]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:31.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:30 smithi184 ceph-mon[100622]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:31.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:30 smithi184 ceph-mon[102844]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:31.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:31.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:31.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:03:33.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:33.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:32 smithi184 ceph-mon[102844]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:33.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:33.125 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:32 smithi184 ceph-mon[100622]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:32 smithi196 ceph-mon[76349]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:35.023 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:03:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:34 smithi196 ceph-mon[76349]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:34 smithi184 ceph-mon[100622]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:34 smithi184 ceph-mon[102844]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 12s ago 5h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:03:35.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 5h 509M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 12s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (4h) 12s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 5h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (4h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:03:35.572 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:03:35.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 12s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:03:35.890 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:03:35.890 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:03:35.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:03:35.891 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:03:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:35 smithi196 ceph-mon[76349]: from='client.75134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:03:36.160 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:03:36.161 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:03:36.161 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:03:36.161 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:03:36.161 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:03:36.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:35 smithi184 ceph-mon[100622]: from='client.75134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:36.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:35 smithi184 ceph-mon[102844]: from='client.75134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:36.475 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:03:36.475 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:36.475 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:36.475 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:36.475 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: from='client.75140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: from='client.65181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4029341099' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: from='client.65193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1505159921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: from='client.75140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: from='client.65181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4029341099' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: from='client.65193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1505159921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: from='client.75140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: from='client.65181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4029341099' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: from='client.65193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1505159921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:38 smithi196 ceph-mon[76349]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:38 smithi184 ceph-mon[100622]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:38 smithi184 ceph-mon[102844]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:40 smithi196 ceph-mon[76349]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:40 smithi184 ceph-mon[100622]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:40 smithi184 ceph-mon[102844]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:41.685 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:41.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:41.686 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:41.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:41.686 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:41.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:03:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[100622]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:42 smithi184 ceph-mon[102844]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:42 smithi196 ceph-mon[76349]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:44 smithi196 ceph-mon[76349]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[100622]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[102844]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:46 smithi196 ceph-mon[76349]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:46 smithi184 ceph-mon[102844]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:46 smithi184 ceph-mon[100622]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:49.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:48 smithi196 ceph-mon[76349]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:48 smithi184 ceph-mon[100622]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:48 smithi184 ceph-mon[102844]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:51.036 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:50 smithi184 ceph-mon[102844]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:51.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:50 smithi184 ceph-mon[100622]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:50 smithi196 ceph-mon[76349]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:51.488Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:51.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:03:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:03:51.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:03:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:03:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:03:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:52 smithi196 ceph-mon[76349]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:52 smithi184 ceph-mon[100622]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:52 smithi184 ceph-mon[102844]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:54 smithi196 ceph-mon[76349]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:54 smithi184 ceph-mon[100622]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:54 smithi184 ceph-mon[102844]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:56 smithi196 ceph-mon[76349]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:56 smithi184 ceph-mon[102844]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:56 smithi184 ceph-mon[100622]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:58 smithi196 ceph-mon[76349]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:58 smithi184 ceph-mon[100622]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:58 smithi184 ceph-mon[102844]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:00 smithi196 ceph-mon[76349]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:00 smithi184 ceph-mon[100622]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:00 smithi184 ceph-mon[102844]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:01.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:01.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:01.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:04:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[102844]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[100622]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:02 smithi196 ceph-mon[76349]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[100622]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:04 smithi184 ceph-mon[102844]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:04 smithi196 ceph-mon[76349]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:06.770 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:04:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:06 smithi184 ceph-mon[100622]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:04:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:06 smithi184 ceph-mon[102844]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:04:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:06 smithi196 ceph-mon[76349]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:04:07.297 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:04:07.297 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T01:04:07.297 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T01:04:07.297 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 3s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 5h 510M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:04:07.298 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 5s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 3s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 5h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 3s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 3s ago 5h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:04:07.299 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 5h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:04:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:04:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (4h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:04:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (4h) 5s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:04:07.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (4h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:04:07.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:04:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:04:07.881 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:04:07.882 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:04:07.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[100622]: from='client.65205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:07.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[100622]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:07.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1607934837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:07.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[102844]: from='client.65205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:07.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[102844]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:07.883 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1607934837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:07 smithi196 ceph-mon[76349]: from='client.65205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:07 smithi196 ceph-mon[76349]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1607934837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:08.191 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:04:08.191 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:08.192 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:08.192 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:08 smithi196 ceph-mon[76349]: from='client.65217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:08 smithi196 ceph-mon[76349]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:08 smithi196 ceph-mon[76349]: from='client.65229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2644011653' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[100622]: from='client.65217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[100622]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[100622]: from='client.65229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2644011653' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[102844]: from='client.65217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[102844]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 331 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[102844]: from='client.65229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2644011653' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:11.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:10 smithi196 ceph-mon[76349]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:11.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:10 smithi184 ceph-mon[100622]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:11.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:10 smithi184 ceph-mon[102844]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:11.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:11.547 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:11.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:11.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:04:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:12 smithi184 ceph-mon[100622]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:04:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:12 smithi184 ceph-mon[102844]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:04:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:12 smithi196 ceph-mon[76349]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 314 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:04:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:14 smithi196 ceph-mon[76349]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T01:04:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:14 smithi184 ceph-mon[100622]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T01:04:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:14 smithi184 ceph-mon[102844]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T01:04:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:16 smithi196 ceph-mon[76349]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T01:04:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:16 smithi184 ceph-mon[100622]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T01:04:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:16 smithi184 ceph-mon[102844]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-21T01:04:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:18 smithi196 ceph-mon[76349]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T01:04:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:18 smithi184 ceph-mon[100622]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T01:04:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:18 smithi184 ceph-mon[102844]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T01:04:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:20 smithi196 ceph-mon[76349]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:04:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:20 smithi184 ceph-mon[100622]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:04:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:20 smithi184 ceph-mon[102844]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:04:21.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:21.490Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:21.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:21.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:21.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:21.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:04:23.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:23.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:22 smithi196 ceph-mon[76349]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:04:23.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[100622]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:04:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[102844]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:24 smithi196 ceph-mon[76349]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-21T01:04:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[100622]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-21T01:04:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:25.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:24 smithi184 ceph-mon[102844]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 138 op/s 2024-09-21T01:04:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:26 smithi196 ceph-mon[76349]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:04:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:26 smithi184 ceph-mon[100622]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:04:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:26 smithi184 ceph-mon[102844]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:04:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:28 smithi196 ceph-mon[76349]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:04:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:28 smithi184 ceph-mon[100622]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:04:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:28 smithi184 ceph-mon[102844]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:04:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:31.095 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:30 smithi196 ceph-mon[76349]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:31.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:30 smithi184 ceph-mon[100622]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:31.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:30 smithi184 ceph-mon[102844]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 306 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:31.490Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:31.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:31.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:32.265 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:04:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:32 smithi196 ceph-mon[76349]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:04:33.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:33.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:32 smithi184 ceph-mon[100622]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:04:33.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:33.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:32 smithi184 ceph-mon[102844]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:04:35.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:34 smithi184 ceph-mon[102844]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:35.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:34 smithi184 ceph-mon[100622]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:34 smithi196 ceph-mon[76349]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:04:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:36 smithi196 ceph-mon[76349]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:04:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:36 smithi184 ceph-mon[100622]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:04:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:36 smithi184 ceph-mon[102844]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:04:38.485 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 15s ago 5h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:04:39.010 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 5h 510M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 15s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 5h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 5h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 5h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 5h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 5h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:04:39.011 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 5h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:04:39.012 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:04:39.012 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:04:39.012 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:04:39.012 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:04:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:38 smithi196 ceph-mon[76349]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T01:04:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:38 smithi184 ceph-mon[100622]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T01:04:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:38 smithi184 ceph-mon[102844]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:04:39.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:04:39.321 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:04:39.321 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:04:39.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:04:39.321 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:04:39.588 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:04:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:04:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:04:39.589 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:04:39.902 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:04:39.902 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:39.902 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:39.903 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:39.903 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:39 smithi196 ceph-mon[76349]: from='client.65241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:39 smithi196 ceph-mon[76349]: from='client.75212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:39 smithi196 ceph-mon[76349]: from='client.75218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3627535136' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[102844]: from='client.65241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[102844]: from='client.75212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[102844]: from='client.75218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3627535136' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[100622]: from='client.65241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[100622]: from='client.75212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[100622]: from='client.75218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3627535136' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:40 smithi196 ceph-mon[76349]: from='client.65265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:40 smithi196 ceph-mon[76349]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:04:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1100540254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[100622]: from='client.65265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[100622]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:04:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1100540254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[102844]: from='client.65265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[102844]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:04:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1100540254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:41.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:41.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:41.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:04:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:43 smithi196 ceph-mon[76349]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T01:04:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[100622]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[102844]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:44 smithi196 ceph-mon[76349]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:04:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[100622]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[102844]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:04:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:46 smithi196 ceph-mon[76349]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T01:04:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:46 smithi184 ceph-mon[100622]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T01:04:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:46 smithi184 ceph-mon[102844]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T01:04:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:48 smithi196 ceph-mon[76349]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:48 smithi184 ceph-mon[100622]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:48 smithi184 ceph-mon[102844]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:04:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:50 smithi196 ceph-mon[76349]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T01:04:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:50 smithi184 ceph-mon[100622]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T01:04:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:50 smithi184 ceph-mon[102844]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T01:04:51.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:51.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:51.687 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:51.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:04:51.688 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:04:51.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:04:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:04:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:04:53.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:53.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:52 smithi184 ceph-mon[102844]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:53.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:53.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:52 smithi184 ceph-mon[100622]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:53.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:52 smithi196 ceph-mon[76349]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:54 smithi196 ceph-mon[76349]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:54 smithi184 ceph-mon[100622]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:54 smithi184 ceph-mon[102844]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:57.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:56 smithi196 ceph-mon[76349]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:56 smithi184 ceph-mon[100622]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:56 smithi184 ceph-mon[102844]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:58 smithi196 ceph-mon[76349]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:58 smithi184 ceph-mon[100622]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:58 smithi184 ceph-mon[102844]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:00 smithi196 ceph-mon[76349]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:00 smithi184 ceph-mon[100622]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:00 smithi184 ceph-mon[102844]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:01.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:01.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:01.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:05:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[102844]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[100622]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:03 smithi196 ceph-mon[76349]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:04 smithi196 ceph-mon[76349]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[102844]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[100622]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:06 smithi196 ceph-mon[76349]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:06 smithi184 ceph-mon[100622]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:06 smithi184 ceph-mon[102844]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:08 smithi196 ceph-mon[76349]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:08 smithi184 ceph-mon[100622]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:08 smithi184 ceph-mon[102844]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:10.197 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:05:10.722 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:05:10.722 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T01:05:10.722 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 8s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 6s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 5h 511M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 8s ago 5h 18.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 6s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 5h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:05:10.723 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 6s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 6s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 6s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 5h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:05:10.724 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:05:11.036 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:10 smithi184 ceph-mon[100622]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:11.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:10 smithi184 ceph-mon[100622]: from='client.65277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:11.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:10 smithi184 ceph-mon[102844]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:11.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:10 smithi184 ceph-mon[102844]: from='client.65277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:11.037 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:05:11.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:05:11.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:05:11.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:11.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:05:11.038 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:05:11.079 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:10 smithi196 ceph-mon[76349]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:11.079 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:10 smithi196 ceph-mon[76349]: from='client.65277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:05:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:05:11.308 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:05:11.308 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:05:11.308 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:05:11.618 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:11.493Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:11.619 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:11.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:11.619 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:11.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:11.619 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:05:11.619 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:11.619 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:11 smithi196 ceph-mon[76349]: from='client.75248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:11 smithi196 ceph-mon[76349]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/191898771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1396086878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:12.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[100622]: from='client.75248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[100622]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/191898771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:12.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1396086878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:12.289 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:05:12.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[102844]: from='client.75248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[102844]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:12.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/191898771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:12.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1396086878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:12 smithi196 ceph-mon[76349]: from='client.65301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:12 smithi196 ceph-mon[76349]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:13.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[100622]: from='client.65301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:13.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:13.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[100622]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:13.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[102844]: from='client.65301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:13.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:13.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:12 smithi184 ceph-mon[102844]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:14 smithi184 ceph-mon[100622]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:14 smithi184 ceph-mon[102844]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:14 smithi196 ceph-mon[76349]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:16 smithi184 ceph-mon[100622]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:16 smithi184 ceph-mon[102844]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:16 smithi196 ceph-mon[76349]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:18 smithi196 ceph-mon[76349]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:18 smithi184 ceph-mon[100622]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:18 smithi184 ceph-mon[102844]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:20 smithi196 ceph-mon[76349]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:20 smithi184 ceph-mon[100622]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:20 smithi184 ceph-mon[102844]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:21.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:21.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:21.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:22 smithi196 ceph-mon[76349]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[100622]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[102844]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:24 smithi196 ceph-mon[76349]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[100622]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[102844]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:26 smithi184 ceph-mon[100622]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:26 smithi184 ceph-mon[102844]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:26 smithi196 ceph-mon[76349]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:28 smithi184 ceph-mon[100622]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:28 smithi184 ceph-mon[102844]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:28 smithi196 ceph-mon[76349]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:30 smithi184 ceph-mon[100622]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:30 smithi184 ceph-mon[102844]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:31.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:30 smithi196 ceph-mon[76349]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:31.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:31.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:31.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:05:33.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:33.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:32 smithi184 ceph-mon[100622]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:33.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:33.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:32 smithi184 ceph-mon[102844]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:32 smithi196 ceph-mon[76349]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:34 smithi184 ceph-mon[100622]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:34 smithi184 ceph-mon[102844]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:34 smithi196 ceph-mon[76349]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:37 smithi184 ceph-mon[102844]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:36 smithi184 ceph-mon[100622]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:36 smithi196 ceph-mon[76349]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:39 smithi184 ceph-mon[102844]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:39 smithi184 ceph-mon[100622]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:39 smithi196 ceph-mon[76349]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:41 smithi184 ceph-mon[100622]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:41 smithi184 ceph-mon[102844]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:41 smithi196 ceph-mon[76349]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:41.496Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:41.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:41.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:41.915 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:05:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:05:42.457 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:05:42.457 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 18s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 5h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:05:42.458 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 5h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:05:42.459 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:05:42.768 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:05:42.769 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:05:42.770 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:05:42.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:05:42.770 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:05:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:05:43.044 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:05:43.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:43.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:43.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='client.65313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='client.75284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/24380545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='client.65313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='client.75284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.046 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/24380545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:43.355 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:05:43.355 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:43.355 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='client.65313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='client.75284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/24380545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[100622]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[100622]: from='client.65337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/886556237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[102844]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[102844]: from='client.65337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/886556237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:44 smithi196 ceph-mon[76349]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:44 smithi196 ceph-mon[76349]: from='client.65337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/886556237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[100622]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[102844]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:45 smithi196 ceph-mon[76349]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:05:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:47 smithi184 ceph-mon[102844]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:47 smithi184 ceph-mon[100622]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:47 smithi196 ceph-mon[76349]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:49 smithi184 ceph-mon[100622]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:49 smithi184 ceph-mon[102844]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:49 smithi196 ceph-mon[76349]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:51 smithi184 ceph-mon[100622]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:51 smithi184 ceph-mon[102844]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:51 smithi196 ceph-mon[76349]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:51.497Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:51.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:05:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:05:51.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:05:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:05:52] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T01:05:53.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:53.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:53 smithi196 ceph-mon[76349]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:53 smithi184 ceph-mon[100622]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:53 smithi184 ceph-mon[102844]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:55 smithi184 ceph-mon[100622]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:55 smithi184 ceph-mon[102844]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:55 smithi196 ceph-mon[76349]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:57 smithi184 ceph-mon[100622]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:57 smithi184 ceph-mon[102844]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:57 smithi196 ceph-mon[76349]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:05:59 smithi184 ceph-mon[100622]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:05:59 smithi184 ceph-mon[102844]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:05:59 smithi196 ceph-mon[76349]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:01 smithi184 ceph-mon[100622]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:01 smithi184 ceph-mon[102844]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:01 smithi196 ceph-mon[76349]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:01.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:01.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:01.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:02] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T01:06:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[100622]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[102844]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:03 smithi196 ceph-mon[76349]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[100622]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[102844]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:05 smithi196 ceph-mon[76349]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:07 smithi184 ceph-mon[100622]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:07 smithi184 ceph-mon[102844]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:07 smithi196 ceph-mon[76349]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:08 smithi184 ceph-mon[100622]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:08 smithi184 ceph-mon[102844]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:08 smithi196 ceph-mon[76349]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:10 smithi196 ceph-mon[76349]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:10 smithi184 ceph-mon[100622]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:10 smithi184 ceph-mon[102844]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:11.499Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:11.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:11.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:12] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T01:06:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:12 smithi196 ceph-mon[76349]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:12 smithi184 ceph-mon[100622]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:12 smithi184 ceph-mon[102844]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:13.655 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 11s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 11s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:06:14.198 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 11s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 11s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 11s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 11s ago 5h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 11s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 11s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 11s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 9s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 9s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 5h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:06:14.199 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 11s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:06:14.200 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:06:14.200 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:06:14.200 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 9s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:06:14.520 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:06:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:06:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:06:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:06:14.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:06:14.793 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:06:14.794 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:06:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: from='client.65349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: from='client.75320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: from='client.75326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2483739692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: from='client.65349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: from='client.75320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: from='client.75326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2483739692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:15.107 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:06:15.107 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:15.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:15.107 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:15.107 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: from='client.65349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: from='client.75320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: from='client.75326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2483739692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:16.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:15 smithi184 ceph-mon[100622]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:16.038 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1368314483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:16.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:15 smithi184 ceph-mon[102844]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:16.038 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1368314483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:15 smithi196 ceph-mon[76349]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1368314483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:16 smithi184 ceph-mon[100622]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:16 smithi184 ceph-mon[102844]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:16 smithi196 ceph-mon[76349]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:18 smithi196 ceph-mon[76349]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:18 smithi184 ceph-mon[100622]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:18 smithi184 ceph-mon[102844]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:20 smithi184 ceph-mon[100622]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:20 smithi184 ceph-mon[102844]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:20 smithi196 ceph-mon[76349]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:21.499Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:21.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:21.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:22] "GET /metrics HTTP/1.1" 200 38953 "" "Prometheus/2.43.0" 2024-09-21T01:06:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[100622]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[102844]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:22 smithi196 ceph-mon[76349]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[102844]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[100622]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:24 smithi196 ceph-mon[76349]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:26 smithi184 ceph-mon[100622]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:26 smithi184 ceph-mon[102844]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:26 smithi196 ceph-mon[76349]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:28 smithi184 ceph-mon[100622]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:28 smithi184 ceph-mon[102844]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:28 smithi196 ceph-mon[76349]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:30 smithi184 ceph-mon[100622]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:30 smithi184 ceph-mon[102844]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:30 smithi196 ceph-mon[76349]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:31.500Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:31.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:31.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:06:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:32 smithi184 ceph-mon[102844]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:32 smithi184 ceph-mon[100622]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:33.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:33.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:32 smithi196 ceph-mon[76349]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:34 smithi184 ceph-mon[100622]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:34 smithi184 ceph-mon[102844]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:35.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:34 smithi196 ceph-mon[76349]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:36 smithi184 ceph-mon[100622]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:36 smithi184 ceph-mon[102844]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:36 smithi196 ceph-mon[76349]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:38 smithi184 ceph-mon[102844]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:38 smithi184 ceph-mon[100622]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:38 smithi196 ceph-mon[76349]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:40 smithi184 ceph-mon[102844]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:40 smithi184 ceph-mon[100622]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:40 smithi196 ceph-mon[76349]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:41.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:41.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:41.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:06:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[100622]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[102844]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:42 smithi196 ceph-mon[76349]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[100622]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[102844]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:44 smithi196 ceph-mon[76349]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:06:45.416 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:06:45.942 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:06:45.942 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T01:06:45.942 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 5h 90.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:06:45.943 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 1s ago 5h 70.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:06:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:06:46.252 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:06:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:06:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:06:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:06:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:06:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:06:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:06:46.262 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:06:46.521 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:06:46.522 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:06:46.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:06:46.830 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:06:46.830 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:46.830 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:46.830 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:46.830 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: from='client.65385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: from='client.75356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3562766841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3952845356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: from='client.65385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: from='client.75356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3562766841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3952845356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: from='client.65385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: from='client.75356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3562766841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3952845356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:47 smithi184 ceph-mon[100622]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:48.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:47 smithi184 ceph-mon[102844]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:47 smithi196 ceph-mon[76349]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:48 smithi184 ceph-mon[102844]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:48 smithi184 ceph-mon[100622]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:48 smithi196 ceph-mon[76349]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:50 smithi184 ceph-mon[102844]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:50 smithi184 ceph-mon[100622]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:50 smithi196 ceph-mon[76349]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:51.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:51.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:06:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:06:51.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:06:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:06:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:06:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:52 smithi184 ceph-mon[100622]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:52 smithi184 ceph-mon[102844]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:52 smithi196 ceph-mon[76349]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:55.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:55 smithi196 ceph-mon[76349]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:54 smithi184 ceph-mon[100622]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:55 smithi184 ceph-mon[102844]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:57 smithi196 ceph-mon[76349]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:57.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:57 smithi184 ceph-mon[100622]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:57.253 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:57 smithi184 ceph-mon[102844]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:06:59 smithi184 ceph-mon[100622]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:06:59 smithi184 ceph-mon[102844]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:06:59 smithi196 ceph-mon[76349]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:01 smithi184 ceph-mon[100622]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:01 smithi184 ceph-mon[102844]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:01 smithi196 ceph-mon[76349]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:01.502Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:01.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:01.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:07:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[102844]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[100622]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:03 smithi196 ceph-mon[76349]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[102844]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[100622]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:05 smithi196 ceph-mon[76349]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:07 smithi184 ceph-mon[102844]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:07 smithi184 ceph-mon[100622]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:07 smithi196 ceph-mon[76349]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:09 smithi184 ceph-mon[100622]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:09 smithi184 ceph-mon[102844]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:09 smithi196 ceph-mon[76349]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:11 smithi184 ceph-mon[100622]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:11 smithi184 ceph-mon[102844]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:11 smithi196 ceph-mon[76349]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:11.503Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:11.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:11.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:07:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:13 smithi184 ceph-mon[100622]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:13 smithi184 ceph-mon[102844]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:13 smithi196 ceph-mon[76349]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:15 smithi196 ceph-mon[76349]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:15 smithi184 ceph-mon[100622]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:15 smithi184 ceph-mon[102844]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:17.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:17 smithi196 ceph-mon[76349]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:17.129 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:07:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:17 smithi184 ceph-mon[100622]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:17 smithi184 ceph-mon[102844]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:17.677 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:07:17.677 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:07:17.678 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 14s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:07:17.679 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:07:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:07:18.001 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:07:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:07:18.277 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:07:18.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:18 smithi184 ceph-mon[100622]: from='client.75386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:18.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/248563273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:18.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:18 smithi184 ceph-mon[102844]: from='client.75386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:18.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/248563273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:18 smithi196 ceph-mon[76349]: from='client.75386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/248563273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:18.590 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:07:18.591 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:18.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:18.591 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:18.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[102844]: from='client.65427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[102844]: from='client.75398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[102844]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1305979421' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[100622]: from='client.65427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[100622]: from='client.75398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[100622]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1305979421' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:19 smithi196 ceph-mon[76349]: from='client.65427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:19 smithi196 ceph-mon[76349]: from='client.75398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:19 smithi196 ceph-mon[76349]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1305979421' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:20 smithi196 ceph-mon[76349]: from='client.75410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:20 smithi184 ceph-mon[100622]: from='client.75410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:20 smithi184 ceph-mon[102844]: from='client.75410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:21 smithi196 ceph-mon[76349]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:21.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:21 smithi184 ceph-mon[100622]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:21.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:21 smithi184 ceph-mon[102844]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:21.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:21.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:21.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:07:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:23 smithi196 ceph-mon[76349]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:23 smithi184 ceph-mon[100622]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:23 smithi184 ceph-mon[102844]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:25 smithi196 ceph-mon[76349]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[102844]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[100622]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:27 smithi196 ceph-mon[76349]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:27 smithi184 ceph-mon[102844]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:27 smithi184 ceph-mon[100622]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:29 smithi196 ceph-mon[76349]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:29 smithi184 ceph-mon[100622]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:29 smithi184 ceph-mon[102844]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:31 smithi196 ceph-mon[76349]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:31.496 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:31 smithi184 ceph-mon[100622]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:31.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:31 smithi184 ceph-mon[102844]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:31.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:31.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:31.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:07:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:33 smithi196 ceph-mon[76349]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:33 smithi184 ceph-mon[100622]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:33 smithi184 ceph-mon[102844]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:35.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:35 smithi196 ceph-mon[76349]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:35 smithi184 ceph-mon[100622]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:35 smithi184 ceph-mon[102844]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:37.131 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:37 smithi184 ceph-mon[100622]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:37.131 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:37 smithi184 ceph-mon[102844]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:37 smithi196 ceph-mon[76349]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:39 smithi184 ceph-mon[100622]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:39 smithi184 ceph-mon[102844]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:39 smithi196 ceph-mon[76349]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:41 smithi196 ceph-mon[76349]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:41.505 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:41 smithi184 ceph-mon[100622]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:41.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:41 smithi184 ceph-mon[102844]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:41.505Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:41.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:41.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:07:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:43 smithi196 ceph-mon[76349]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:43 smithi184 ceph-mon[100622]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:43 smithi184 ceph-mon[102844]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[100622]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[102844]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:45 smithi196 ceph-mon[76349]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:07:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:47 smithi184 ceph-mon[100622]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:47 smithi184 ceph-mon[102844]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:47 smithi196 ceph-mon[76349]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:48.886 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:07:49.411 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 5h - - 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 4s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 4s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 6s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 6s ago 5h 514M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 4s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 6s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:07:49.412 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 4s ago 5h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 4s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 4s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 4s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 4s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 4s ago 5h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:07:49.413 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 4s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:07:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:49 smithi184 ceph-mon[100622]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:49.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:49 smithi184 ceph-mon[102844]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:49 smithi196 ceph-mon[76349]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:49.722 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:07:49.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:07:49.724 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:07:49.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:07:49.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:07:49.724 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:07:49.992 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:07:49.993 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:07:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[102844]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[102844]: from='client.75428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2601613149' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[100622]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[100622]: from='client.75428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2601613149' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:50.300 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:07:50.300 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:50.300 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:50.301 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:50 smithi196 ceph-mon[76349]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:50 smithi196 ceph-mon[76349]: from='client.75428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2601613149' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:51.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[102844]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[102844]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:51.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[102844]: from='client.65481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1223086312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:51.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[100622]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[100622]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:51.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[100622]: from='client.65481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1223086312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:51 smithi196 ceph-mon[76349]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:51 smithi196 ceph-mon[76349]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:51 smithi196 ceph-mon[76349]: from='client.65481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1223086312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:51.506Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:51.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:07:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:07:51.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:07:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:07:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:07:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:53 smithi184 ceph-mon[100622]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:53 smithi184 ceph-mon[102844]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:53 smithi196 ceph-mon[76349]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:55 smithi184 ceph-mon[100622]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:55 smithi184 ceph-mon[102844]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:55 smithi196 ceph-mon[76349]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:57.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:57 smithi196 ceph-mon[76349]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:57 smithi184 ceph-mon[100622]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:57 smithi184 ceph-mon[102844]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:59.461 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:59 smithi184 ceph-mon[100622]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:59.461 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:59.461 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:59 smithi184 ceph-mon[102844]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:59.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:59 smithi196 ceph-mon[76349]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:01.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:01 smithi184 ceph-mon[102844]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:01.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:01 smithi184 ceph-mon[100622]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:01 smithi196 ceph-mon[76349]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:01.507Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:01.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:01.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:08:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:03 smithi184 ceph-mon[100622]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:03 smithi184 ceph-mon[102844]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:03 smithi196 ceph-mon[76349]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:04 smithi196 ceph-mon[76349]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[100622]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:04 smithi184 ceph-mon[102844]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:07 smithi184 ceph-mon[100622]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:07.454 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:07 smithi184 ceph-mon[102844]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:07.454 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:07 smithi196 ceph-mon[76349]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:09 smithi184 ceph-mon[100622]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:09 smithi184 ceph-mon[102844]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:09 smithi196 ceph-mon[76349]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:11 smithi184 ceph-mon[100622]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:11 smithi184 ceph-mon[102844]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:11 smithi196 ceph-mon[76349]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:11.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:11.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:11.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:08:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:13 smithi184 ceph-mon[100622]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:13 smithi184 ceph-mon[102844]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:13 smithi196 ceph-mon[76349]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:08:13.812886139Z level=info msg="Completed cleanup jobs" duration=13.437487ms 2024-09-21T01:08:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:15 smithi184 ceph-mon[100622]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:15 smithi184 ceph-mon[102844]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:15 smithi196 ceph-mon[76349]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:17 smithi184 ceph-mon[100622]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:17 smithi184 ceph-mon[102844]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:17 smithi196 ceph-mon[76349]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:19 smithi184 ceph-mon[102844]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:19 smithi184 ceph-mon[100622]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:19 smithi196 ceph-mon[76349]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:20.605 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:08:21.174 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 5h 515M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 16s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:08:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:08:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 16s ago 5h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:08:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:08:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:08:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:08:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:08:21.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:21 smithi184 ceph-mon[100622]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:21.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:21 smithi184 ceph-mon[102844]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:21 smithi196 ceph-mon[76349]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:08:21.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:08:21.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:21.490 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:08:21.490 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:08:21.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:08:21.490 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:08:21.758 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:08:21.759 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:08:21.760 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:21.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:21.760 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:21.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:21.760 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:21.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:08:22.072 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:08:22.072 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:22.072 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:22.072 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:22.072 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:22 smithi196 ceph-mon[76349]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:22 smithi196 ceph-mon[76349]: from='client.65499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:22 smithi196 ceph-mon[76349]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2600564858' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[100622]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[100622]: from='client.65499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[100622]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2600564858' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[102844]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[102844]: from='client.65499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[102844]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2600564858' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:23 smithi196 ceph-mon[76349]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:23 smithi196 ceph-mon[76349]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/829266243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:23.461 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[100622]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[100622]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/829266243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[102844]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[102844]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:23.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/829266243' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[100622]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:24 smithi184 ceph-mon[102844]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:24 smithi196 ceph-mon[76349]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:27 smithi184 ceph-mon[100622]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:27 smithi184 ceph-mon[102844]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:27 smithi196 ceph-mon[76349]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:29 smithi184 ceph-mon[100622]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:29 smithi184 ceph-mon[102844]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:29 smithi196 ceph-mon[76349]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:31.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:31 smithi184 ceph-mon[100622]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:31.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:31 smithi184 ceph-mon[102844]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:31 smithi196 ceph-mon[76349]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:31.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:31.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:31.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:08:33.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:33.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:33 smithi184 ceph-mon[100622]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:33.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:33.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:33 smithi184 ceph-mon[102844]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:33 smithi196 ceph-mon[76349]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:35 smithi184 ceph-mon[100622]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:35 smithi184 ceph-mon[102844]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:35 smithi196 ceph-mon[76349]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:37 smithi196 ceph-mon[76349]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:37 smithi184 ceph-mon[100622]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:37 smithi184 ceph-mon[102844]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:39.302 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:39 smithi184 ceph-mon[100622]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:39.302 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:39 smithi184 ceph-mon[102844]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:39.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:39 smithi196 ceph-mon[76349]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:41.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:41 smithi184 ceph-mon[100622]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:41.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:41 smithi184 ceph-mon[102844]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:41 smithi196 ceph-mon[76349]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:41.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:41.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:41.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:08:43.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:43.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:43 smithi196 ceph-mon[76349]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:43 smithi184 ceph-mon[102844]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:43 smithi184 ceph-mon[100622]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[102844]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[100622]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:44 smithi196 ceph-mon[76349]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:08:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:47 smithi184 ceph-mon[100622]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:47 smithi184 ceph-mon[102844]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:47 smithi196 ceph-mon[76349]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:48 smithi184 ceph-mon[100622]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:48 smithi184 ceph-mon[102844]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:48 smithi196 ceph-mon[76349]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:50 smithi184 ceph-mon[100622]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:50 smithi184 ceph-mon[102844]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:50 smithi196 ceph-mon[76349]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:51.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:51.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:08:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:08:51.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:08:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:08:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:08:52.369 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 5h - - 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 5h - - 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 7s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 7s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 9s ago 5h 516M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 7s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:08:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 9s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 7s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 7s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 7s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 7s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 7s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 7s ago 5h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:08:52.900 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:08:52.901 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 7s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:08:53.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:08:53.210 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:08:53.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:53.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:52 smithi184 ceph-mon[100622]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:53.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:53.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:52 smithi184 ceph-mon[102844]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:52 smithi196 ceph-mon[76349]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:08:53.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:08:53.478 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:08:53.789 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:08:53.790 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:53.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:53.790 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:53.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[102844]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[102844]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[102844]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3210089866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1417845708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[100622]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[100622]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[100622]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3210089866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1417845708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:53 smithi196 ceph-mon[76349]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:53 smithi196 ceph-mon[76349]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:53 smithi196 ceph-mon[76349]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3210089866' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1417845708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:54 smithi196 ceph-mon[76349]: from='client.65553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:54 smithi196 ceph-mon[76349]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:54 smithi184 ceph-mon[102844]: from='client.65553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:54 smithi184 ceph-mon[102844]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:54 smithi184 ceph-mon[100622]: from='client.65553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:54 smithi184 ceph-mon[100622]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:56 smithi184 ceph-mon[100622]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:56 smithi184 ceph-mon[102844]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:56 smithi196 ceph-mon[76349]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:58 smithi184 ceph-mon[102844]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:58 smithi184 ceph-mon[100622]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:59.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:58 smithi196 ceph-mon[76349]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:00.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:00 smithi196 ceph-mon[76349]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:00 smithi184 ceph-mon[100622]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:00 smithi184 ceph-mon[102844]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:01.551 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:01.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:01.551 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:01.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:01.551 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:01.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:09:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:02 smithi184 ceph-mon[100622]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:02 smithi184 ceph-mon[102844]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:02 smithi196 ceph-mon[76349]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[100622]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:04 smithi184 ceph-mon[102844]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:04 smithi196 ceph-mon[76349]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[100622]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:06 smithi184 ceph-mon[102844]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:06 smithi196 ceph-mon[76349]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:08 smithi184 ceph-mon[100622]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:08 smithi184 ceph-mon[102844]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:08 smithi196 ceph-mon[76349]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:10 smithi184 ceph-mon[100622]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:10 smithi184 ceph-mon[102844]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:10 smithi196 ceph-mon[76349]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:11.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:11.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:11.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:09:13.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:13.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:12 smithi196 ceph-mon[76349]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:12 smithi184 ceph-mon[102844]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:12 smithi184 ceph-mon[100622]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:14 smithi184 ceph-mon[100622]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:14 smithi184 ceph-mon[102844]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:14 smithi196 ceph-mon[76349]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:16 smithi184 ceph-mon[100622]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:16 smithi184 ceph-mon[102844]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:16 smithi196 ceph-mon[76349]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:19.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:18 smithi196 ceph-mon[76349]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:18 smithi184 ceph-mon[100622]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:18 smithi184 ceph-mon[102844]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:20 smithi196 ceph-mon[76349]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:20 smithi184 ceph-mon[100622]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:20 smithi184 ceph-mon[102844]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:21.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:21.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:21.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:21.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:21.726 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:21.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:09:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:22 smithi184 ceph-mon[100622]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:22 smithi184 ceph-mon[102844]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:22 smithi196 ceph-mon[76349]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:24.085 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:09:24.610 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 5h - - 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 5h 516M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 19s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:09:24.611 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:09:24.612 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:09:24.922 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:09:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:09:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:09:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:09:24.923 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[100622]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[100622]: from='client.65565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[102844]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:24.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:24 smithi184 ceph-mon[102844]: from='client.65565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:24 smithi196 ceph-mon[76349]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:24 smithi196 ceph-mon[76349]: from='client.65565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:25.193 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:09:25.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:09:25.511 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:09:25.512 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:25.512 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:25.512 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:25.512 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='client.65571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3509357392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='client.75554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/620472716' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='client.65571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3509357392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='client.75554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/620472716' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='client.65571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3509357392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='client.75554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/620472716' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:26 smithi196 ceph-mon[76349]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:26 smithi184 ceph-mon[100622]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:26 smithi184 ceph-mon[102844]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:28 smithi184 ceph-mon[100622]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:28 smithi184 ceph-mon[102844]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:28 smithi196 ceph-mon[76349]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:30 smithi184 ceph-mon[100622]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:30 smithi184 ceph-mon[102844]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:30 smithi196 ceph-mon[76349]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:31.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:31.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:31.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:09:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:32 smithi184 ceph-mon[100622]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:32 smithi184 ceph-mon[102844]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:32 smithi196 ceph-mon[76349]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:34 smithi184 ceph-mon[100622]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:34 smithi184 ceph-mon[102844]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:34 smithi196 ceph-mon[76349]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:36 smithi184 ceph-mon[100622]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:36 smithi184 ceph-mon[102844]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:36 smithi196 ceph-mon[76349]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:39.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:38 smithi196 ceph-mon[76349]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:38 smithi184 ceph-mon[100622]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:38 smithi184 ceph-mon[102844]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:40 smithi184 ceph-mon[100622]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:40 smithi184 ceph-mon[102844]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:40 smithi196 ceph-mon[76349]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:41.603 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:41.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:41.603 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:41.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:41.603 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:41.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:09:43.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:43.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:42 smithi184 ceph-mon[100622]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:43.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:43.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:42 smithi184 ceph-mon[102844]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:42 smithi196 ceph-mon[76349]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[102844]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[100622]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:44 smithi196 ceph-mon[76349]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[102844]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:46 smithi184 ceph-mon[100622]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:09:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:46 smithi196 ceph-mon[76349]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:48 smithi184 ceph-mon[102844]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:48 smithi184 ceph-mon[100622]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:48 smithi196 ceph-mon[76349]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:50 smithi184 ceph-mon[100622]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:50 smithi184 ceph-mon[102844]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:50 smithi196 ceph-mon[76349]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:51.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:51.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:09:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:09:51.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:09:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:09:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:09:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:52 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:52 smithi184 ceph-mon[102844]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:52 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:52 smithi184 ceph-mon[100622]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:52 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:52 smithi196 ceph-mon[76349]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:54 smithi184 ceph-mon[100622]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:54 smithi184 ceph-mon[102844]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:54 smithi196 ceph-mon[76349]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:55.808 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:09:56.339 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:09:56.339 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 5h - - 2024-09-21T01:09:56.339 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T01:09:56.339 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:09:56.339 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 5h 516M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 10s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 10s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 10s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:09:56.340 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 10s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 10s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 10s ago 5h 66.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:09:56.341 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 10s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:09:56.651 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:09:56.921 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:09:56.922 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:09:57.234 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:09:57.234 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:57.234 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[100622]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[100622]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[100622]: from='client.75572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1338175940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[102844]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[102844]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[102844]: from='client.75572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.235 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1338175940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:56 smithi196 ceph-mon[76349]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:56 smithi196 ceph-mon[76349]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:56 smithi196 ceph-mon[76349]: from='client.75572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1338175940' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[100622]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[100622]: from='client.65625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1708382384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[102844]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[102844]: from='client.65625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1708382384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:57 smithi196 ceph-mon[76349]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:57 smithi196 ceph-mon[76349]: from='client.65625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1708382384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:58 smithi184 ceph-mon[100622]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:58 smithi184 ceph-mon[102844]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:58 smithi196 ceph-mon[76349]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:00.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:01.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:00 smithi196 ceph-mon[76349]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:01.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:00 smithi184 ceph-mon[100622]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:00 smithi184 ceph-mon[102844]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:01.780 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:01.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:01.780 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:01.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:01.780 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:01.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:10:03.018 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:02 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:03.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:02 smithi184 ceph-mon[100622]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:03.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:02 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:03.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:02 smithi184 ceph-mon[102844]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:02 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:02 smithi196 ceph-mon[76349]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:05 smithi196 ceph-mon[76349]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[100622]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[102844]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[100622]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:06 smithi184 ceph-mon[102844]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:06 smithi196 ceph-mon[76349]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:08 smithi184 ceph-mon[100622]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:08 smithi184 ceph-mon[102844]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:08 smithi196 ceph-mon[76349]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:10 smithi184 ceph-mon[100622]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:10 smithi184 ceph-mon[102844]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:10 smithi196 ceph-mon[76349]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:11.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:11.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:11.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:10:13.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:13.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:12 smithi196 ceph-mon[76349]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:12 smithi184 ceph-mon[100622]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:12 smithi184 ceph-mon[102844]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:14 smithi184 ceph-mon[100622]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:14 smithi184 ceph-mon[102844]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:14 smithi196 ceph-mon[76349]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:16 smithi184 ceph-mon[100622]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:16 smithi184 ceph-mon[102844]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:16 smithi196 ceph-mon[76349]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:18 smithi184 ceph-mon[102844]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:18 smithi184 ceph-mon[100622]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:18 smithi196 ceph-mon[76349]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:20 smithi184 ceph-mon[100622]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:20 smithi184 ceph-mon[102844]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:20 smithi196 ceph-mon[76349]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:21.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:21.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:21.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:22] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:10:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:22 smithi184 ceph-mon[100622]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:22 smithi184 ceph-mon[102844]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:22 smithi196 ceph-mon[76349]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[102844]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[100622]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:25 smithi196 ceph-mon[76349]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[100622]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[102844]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:26 smithi196 ceph-mon[76349]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:27.531 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:10:28.063 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:10:28.063 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T01:10:28.063 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 5h - - 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 516M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:10:28.064 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 67.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:10:28.065 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:10:28.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:10:28.374 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:10:28.374 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:10:28.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:10:28.374 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:10:28.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:10:28.640 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:10:28.640 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:10:28.640 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:10:28.641 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:10:28.954 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:28.954 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:28.955 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:28.955 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:28.955 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[102844]: from='client.75602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[102844]: from='client.65643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[102844]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[102844]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4264582159' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[100622]: from='client.75602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[100622]: from='client.65643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[100622]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[100622]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4264582159' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:28 smithi196 ceph-mon[76349]: from='client.75602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:28 smithi196 ceph-mon[76349]: from='client.65643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:28 smithi196 ceph-mon[76349]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:28 smithi196 ceph-mon[76349]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4264582159' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[102844]: from='client.75626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2869419281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[100622]: from='client.75626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2869419281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:29 smithi196 ceph-mon[76349]: from='client.75626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2869419281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:30 smithi184 ceph-mon[102844]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:30 smithi184 ceph-mon[100622]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:30 smithi196 ceph-mon[76349]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:31.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:31.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:31.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:10:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:32 smithi184 ceph-mon[102844]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:32 smithi184 ceph-mon[100622]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:32 smithi196 ceph-mon[76349]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:34 smithi184 ceph-mon[100622]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:34 smithi184 ceph-mon[102844]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:34 smithi196 ceph-mon[76349]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:36 smithi184 ceph-mon[100622]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:36 smithi184 ceph-mon[102844]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:36 smithi196 ceph-mon[76349]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:38 smithi184 ceph-mon[100622]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:38 smithi184 ceph-mon[102844]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:38 smithi196 ceph-mon[76349]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:41.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:40 smithi196 ceph-mon[76349]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:40 smithi184 ceph-mon[100622]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:40 smithi184 ceph-mon[102844]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:41.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:41.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:41.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:10:43.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:43.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:43 smithi196 ceph-mon[76349]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:43.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:43.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:43 smithi184 ceph-mon[100622]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:43.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:43.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:43 smithi184 ceph-mon[102844]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[100622]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[102844]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:45 smithi196 ceph-mon[76349]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[100622]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[102844]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:47 smithi196 ceph-mon[76349]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:10:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:49 smithi184 ceph-mon[100622]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:49 smithi184 ceph-mon[102844]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:49 smithi196 ceph-mon[76349]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:51 smithi184 ceph-mon[100622]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:51 smithi184 ceph-mon[102844]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:51 smithi196 ceph-mon[76349]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:51.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:51.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:10:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:10:51.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:10:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:10:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:10:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:53 smithi184 ceph-mon[100622]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:53 smithi184 ceph-mon[102844]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:53 smithi196 ceph-mon[76349]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:55 smithi184 ceph-mon[100622]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:55 smithi184 ceph-mon[102844]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:55 smithi196 ceph-mon[76349]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:57 smithi184 ceph-mon[100622]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:57 smithi184 ceph-mon[102844]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:57 smithi196 ceph-mon[76349]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:59.252 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:10:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:10:59 smithi184 ceph-mon[100622]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:10:59 smithi184 ceph-mon[102844]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:10:59 smithi196 ceph-mon[76349]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:59.778 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 5h 517M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:10:59.779 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 67.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:10:59.780 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:11:00.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:11:00.094 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:11:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:00 smithi184 ceph-mon[100622]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:00 smithi184 ceph-mon[102844]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:11:00.364 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:11:00.365 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:11:00.365 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:11:00.365 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:11:00.365 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:11:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:00 smithi196 ceph-mon[76349]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:00.675 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:11:00.675 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:00.675 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:00.675 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:00.675 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:01 smithi196 ceph-mon[76349]: from='client.75644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:01 smithi196 ceph-mon[76349]: from='client.65685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:01 smithi196 ceph-mon[76349]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1552928042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2002087055' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[100622]: from='client.75644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[100622]: from='client.65685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[100622]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1552928042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2002087055' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[102844]: from='client.75644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[102844]: from='client.65685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[102844]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1552928042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:01.524 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2002087055' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:01.523Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:01.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:01.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:02 smithi184 ceph-mon[100622]: from='client.65697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:02 smithi184 ceph-mon[102844]: from='client.65697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:11:02.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:02 smithi196 ceph-mon[76349]: from='client.65697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:03.333 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:03.333 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:03 smithi196 ceph-mon[76349]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:03.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:03.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:03 smithi184 ceph-mon[100622]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:03.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:03.475 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:03 smithi184 ceph-mon[102844]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:05 smithi196 ceph-mon[76349]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.143 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[100622]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:05.143 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.143 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.143 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[102844]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:05.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:05.144 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[100622]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[102844]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:07 smithi196 ceph-mon[76349]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:09 smithi184 ceph-mon[100622]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:09 smithi184 ceph-mon[102844]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:09.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:09 smithi196 ceph-mon[76349]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:11.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:11 smithi184 ceph-mon[100622]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:11.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:11 smithi184 ceph-mon[102844]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:11 smithi196 ceph-mon[76349]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:11.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:11.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:11.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:11:13.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:13.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:13 smithi196 ceph-mon[76349]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:13 smithi184 ceph-mon[100622]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:13 smithi184 ceph-mon[102844]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:15 smithi184 ceph-mon[100622]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:15 smithi184 ceph-mon[102844]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:15.553 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:15 smithi196 ceph-mon[76349]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:17 smithi184 ceph-mon[100622]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:17 smithi184 ceph-mon[102844]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:17 smithi196 ceph-mon[76349]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:19 smithi184 ceph-mon[100622]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:19 smithi184 ceph-mon[102844]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:19 smithi196 ceph-mon[76349]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:21.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:21 smithi184 ceph-mon[100622]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:21.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:21 smithi184 ceph-mon[102844]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:21 smithi196 ceph-mon[76349]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:21.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:21.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:21.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:11:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:23 smithi184 ceph-mon[100622]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:23 smithi184 ceph-mon[102844]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:23 smithi196 ceph-mon[76349]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[100622]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[102844]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:25 smithi196 ceph-mon[76349]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[100622]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[102844]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:27 smithi196 ceph-mon[76349]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:29 smithi184 ceph-mon[100622]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:29 smithi184 ceph-mon[102844]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:29 smithi196 ceph-mon[76349]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:30.972 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:11:31.501 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:11:31.501 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T01:11:31.501 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 93.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 6s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 6s ago 5h 518M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 6s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:11:31.502 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 68.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:11:31.503 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:11:31.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:31 smithi184 ceph-mon[100622]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:31.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:31 smithi184 ceph-mon[102844]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:31 smithi196 ceph-mon[76349]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:31.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:31.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:31.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:31.815 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:11:31.815 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:11:31.816 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:11:32.088 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:11:32.089 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:11:32.089 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:11:32.340 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[100622]: from='client.75674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[100622]: from='client.75680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1187204451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:32.341 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[102844]: from='client.75674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.341 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[102844]: from='client.75680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.341 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1187204451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:32.415 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:11:32.415 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:32.415 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:32 smithi196 ceph-mon[76349]: from='client.75674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:32 smithi196 ceph-mon[76349]: from='client.75680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1187204451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[102844]: from='client.75686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[102844]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[102844]: from='client.65733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3573629187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[100622]: from='client.75686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[100622]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[100622]: from='client.65733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3573629187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:33 smithi196 ceph-mon[76349]: from='client.75686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:33 smithi196 ceph-mon[76349]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:33 smithi196 ceph-mon[76349]: from='client.65733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3573629187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:35 smithi184 ceph-mon[100622]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:35 smithi184 ceph-mon[102844]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:35 smithi196 ceph-mon[76349]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:37 smithi184 ceph-mon[100622]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:37 smithi184 ceph-mon[102844]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:37 smithi196 ceph-mon[76349]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:38 smithi184 ceph-mon[100622]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:38 smithi184 ceph-mon[102844]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:38 smithi196 ceph-mon[76349]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:40 smithi184 ceph-mon[100622]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:40 smithi184 ceph-mon[102844]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:40 smithi196 ceph-mon[76349]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:41.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:41.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:41.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:11:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:42 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:42 smithi184 ceph-mon[100622]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:42 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:42 smithi184 ceph-mon[102844]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:43.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:42 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:43.350 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:42 smithi196 ceph-mon[76349]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:44.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[100622]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:44.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:44.993 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[102844]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:44.994 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:45.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:44 smithi196 ceph-mon[76349]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:45.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:45.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:45.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:45.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[100622]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:47.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[102844]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:46 smithi196 ceph-mon[76349]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:11:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:48 smithi184 ceph-mon[100622]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:48 smithi184 ceph-mon[102844]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:48 smithi196 ceph-mon[76349]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:50 smithi184 ceph-mon[100622]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:50 smithi184 ceph-mon[102844]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:50 smithi196 ceph-mon[76349]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:51.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:51.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:11:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:11:51.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:11:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:11:52] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:11:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:53 smithi184 ceph-mon[100622]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:53 smithi184 ceph-mon[102844]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:53 smithi196 ceph-mon[76349]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:55.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:55 smithi196 ceph-mon[76349]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:55 smithi184 ceph-mon[100622]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:55 smithi184 ceph-mon[102844]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:57 smithi184 ceph-mon[100622]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:57 smithi184 ceph-mon[102844]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:57 smithi196 ceph-mon[76349]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:11:59 smithi184 ceph-mon[100622]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:11:59 smithi184 ceph-mon[102844]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:11:59 smithi196 ceph-mon[76349]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:01 smithi184 ceph-mon[100622]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:01 smithi184 ceph-mon[102844]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:01 smithi196 ceph-mon[76349]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:01.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:01.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:01.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:12:02.709 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:12:03.246 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 18s ago 5h 519M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 16s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 18s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 18s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 18s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 18s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:12:03.247 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 16s ago 5h 68.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:12:03.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:12:03.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:12:03.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:12:03.248 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:12:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:03 smithi184 ceph-mon[100622]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:03 smithi184 ceph-mon[102844]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:03 smithi196 ceph-mon[76349]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:03.554 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:12:03.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:12:03.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:03.556 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:12:03.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:12:03.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:12:03.556 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:12:03.822 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:12:03.823 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:12:04.138 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:12:04.238 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:04.239 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:04.239 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:04.239 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:04.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[100622]: from='client.75710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[100622]: from='client.75716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[100622]: from='client.75722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/955543317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:04.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[102844]: from='client.75710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[102844]: from='client.75716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[102844]: from='client.75722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/955543317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:04.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:04 smithi196 ceph-mon[76349]: from='client.75710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:04 smithi196 ceph-mon[76349]: from='client.75716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:04 smithi196 ceph-mon[76349]: from='client.75722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:04.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/955543317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: from='client.75734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/585194752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: from='client.75734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/585194752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: from='client.75734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/585194752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[100622]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[102844]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:07 smithi196 ceph-mon[76349]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:09 smithi184 ceph-mon[100622]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:09 smithi184 ceph-mon[102844]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:09.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:09 smithi196 ceph-mon[76349]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:11 smithi184 ceph-mon[100622]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:11 smithi184 ceph-mon[102844]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:11 smithi196 ceph-mon[76349]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:11.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:11.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:11.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:12:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:13 smithi196 ceph-mon[76349]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:13 smithi184 ceph-mon[102844]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:13 smithi184 ceph-mon[100622]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:15 smithi196 ceph-mon[76349]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:15 smithi184 ceph-mon[102844]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:15 smithi184 ceph-mon[100622]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:17 smithi196 ceph-mon[76349]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:17 smithi184 ceph-mon[100622]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:17 smithi184 ceph-mon[102844]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:19 smithi196 ceph-mon[76349]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:19 smithi184 ceph-mon[100622]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:19 smithi184 ceph-mon[102844]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:21 smithi196 ceph-mon[76349]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:21.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:21 smithi184 ceph-mon[100622]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:21.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:21 smithi184 ceph-mon[102844]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:21.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:21.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:21.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:22] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:12:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:23 smithi196 ceph-mon[76349]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:12:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:23 smithi184 ceph-mon[102844]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:12:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:23 smithi184 ceph-mon[100622]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:12:25.231 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:25 smithi196 ceph-mon[76349]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:25.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[102844]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[100622]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:27 smithi196 ceph-mon[76349]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[102844]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:27.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.509 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[100622]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:27.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:27.510 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:29 smithi196 ceph-mon[76349]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:29.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:29 smithi184 ceph-mon[100622]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:29.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:29 smithi184 ceph-mon[102844]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:29.526 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:31 smithi196 ceph-mon[76349]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:31.530 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:31 smithi184 ceph-mon[100622]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:31.530 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:31 smithi184 ceph-mon[102844]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:31.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:31.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:31.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:12:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:33 smithi196 ceph-mon[76349]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:33 smithi184 ceph-mon[100622]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:33 smithi184 ceph-mon[102844]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:34.433 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:12:34.960 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:12:34.960 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 5h - - 2024-09-21T01:12:34.960 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 5h - - 2024-09-21T01:12:34.960 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 9s ago 5h 517M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 9s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:12:34.961 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 65.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:12:35.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:12:35.273 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:12:35.273 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:35 smithi184 ceph-mon[100622]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:35.274 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:35 smithi184 ceph-mon[102844]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:35 smithi196 ceph-mon[76349]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:12:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:12:35.543 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:12:35.543 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:12:35.543 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:12:35.849 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:12:35.849 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:35.849 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[100622]: from='client.65781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[100622]: from='client.75752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[100622]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/523104440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2243615663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[102844]: from='client.65781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[102844]: from='client.75752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[102844]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/523104440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2243615663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:36 smithi196 ceph-mon[76349]: from='client.65781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:36 smithi196 ceph-mon[76349]: from='client.75752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:36 smithi196 ceph-mon[76349]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/523104440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2243615663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:37 smithi196 ceph-mon[76349]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:37 smithi196 ceph-mon[76349]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:37 smithi184 ceph-mon[100622]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:37 smithi184 ceph-mon[100622]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:37 smithi184 ceph-mon[102844]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:37 smithi184 ceph-mon[102844]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:39 smithi184 ceph-mon[100622]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:39 smithi184 ceph-mon[102844]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:39 smithi196 ceph-mon[76349]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:41 smithi196 ceph-mon[76349]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:41.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:41 smithi184 ceph-mon[100622]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:41.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:41 smithi184 ceph-mon[102844]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:41.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:41.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:41.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:12:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:43 smithi196 ceph-mon[76349]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:43 smithi184 ceph-mon[100622]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:43 smithi184 ceph-mon[102844]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:45 smithi196 ceph-mon[76349]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:45 smithi184 ceph-mon[100622]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:45 smithi184 ceph-mon[102844]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:46.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.182 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:47 smithi196 ceph-mon[76349]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:47.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[100622]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[102844]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:47.418 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:12:49.247 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:49 smithi184 ceph-mon[100622]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:49.248 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:49 smithi184 ceph-mon[102844]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:49 smithi196 ceph-mon[76349]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:51.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:51 smithi184 ceph-mon[100622]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:51.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:51 smithi184 ceph-mon[102844]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:51 smithi196 ceph-mon[76349]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:51.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:51.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:12:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:12:51.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:12:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:12:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:12:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:53 smithi184 ceph-mon[100622]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:53 smithi184 ceph-mon[102844]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:53 smithi196 ceph-mon[76349]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:55 smithi184 ceph-mon[102844]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:55 smithi184 ceph-mon[100622]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:55 smithi196 ceph-mon[76349]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:57 smithi184 ceph-mon[100622]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:57 smithi184 ceph-mon[102844]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:57 smithi196 ceph-mon[76349]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:59 smithi184 ceph-mon[100622]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:59 smithi184 ceph-mon[102844]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:59 smithi196 ceph-mon[76349]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:01.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:01 smithi184 ceph-mon[100622]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:01.532 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:01 smithi184 ceph-mon[102844]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:01 smithi196 ceph-mon[76349]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:01.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:01.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:01.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:13:03.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:03.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:03 smithi184 ceph-mon[100622]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:03.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:03 smithi184 ceph-mon[102844]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:03 smithi196 ceph-mon[76349]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:05 smithi184 ceph-mon[100622]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:05 smithi184 ceph-mon[102844]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:05 smithi196 ceph-mon[76349]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:06.146 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:13:06.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 5h - - 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 5h - - 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 1s ago 5h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 1s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:13:06.672 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 1s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 1s ago 5h 518M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 19s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 1s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 1s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 1s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 1s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:13:06.673 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 65.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:13:06.674 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:13:06.674 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:13:06.674 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:13:06.674 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:13:06.983 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:13:06.983 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:13:06.983 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:13:06.984 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:13:07.251 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:13:07.251 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:13:07.251 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:13:07.252 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:13:07.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:07.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:07.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1870368876' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1870368876' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1870368876' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:07.562 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:13:07.562 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:07.562 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:07.562 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:07.562 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[100622]: from='client.75788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[100622]: from='client.65829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[100622]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/414392041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[102844]: from='client.75788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[102844]: from='client.65829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[102844]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/414392041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:08 smithi196 ceph-mon[76349]: from='client.75788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:08 smithi196 ceph-mon[76349]: from='client.65829 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:08 smithi196 ceph-mon[76349]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/414392041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:09.436 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:09 smithi184 ceph-mon[100622]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:09.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:09 smithi184 ceph-mon[102844]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:09 smithi196 ceph-mon[76349]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:11.533 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:11 smithi184 ceph-mon[100622]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:11.533 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:11 smithi184 ceph-mon[102844]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:11.533 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:11.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:11.533 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:11.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:11 smithi196 ceph-mon[76349]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:11.786 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:11.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:13:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:13 smithi184 ceph-mon[100622]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:13 smithi184 ceph-mon[102844]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:13 smithi196 ceph-mon[76349]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:15 smithi184 ceph-mon[100622]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:15 smithi184 ceph-mon[102844]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:15 smithi196 ceph-mon[76349]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:17 smithi196 ceph-mon[76349]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:17 smithi184 ceph-mon[100622]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:17 smithi184 ceph-mon[102844]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:19 smithi196 ceph-mon[76349]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:19 smithi184 ceph-mon[100622]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:19 smithi184 ceph-mon[102844]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:21 smithi196 ceph-mon[76349]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:21.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:21 smithi184 ceph-mon[100622]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:21 smithi184 ceph-mon[102844]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:21.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:21.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:21.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:13:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:22 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:22 smithi196 ceph-mon[76349]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:22 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:22 smithi184 ceph-mon[100622]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:22 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:22 smithi184 ceph-mon[102844]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:25 smithi184 ceph-mon[100622]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:25 smithi184 ceph-mon[102844]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:25 smithi196 ceph-mon[76349]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:26 smithi196 ceph-mon[76349]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[100622]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:26 smithi184 ceph-mon[102844]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:28.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:29 smithi184 ceph-mon[100622]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:29.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:29 smithi184 ceph-mon[102844]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:29 smithi196 ceph-mon[76349]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:31.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:31 smithi184 ceph-mon[100622]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:31.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:31 smithi184 ceph-mon[102844]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:31 smithi196 ceph-mon[76349]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:31.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:31.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:31.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:13:33.315 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:33.315 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:33 smithi184 ceph-mon[100622]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:33.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:33.315 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:33 smithi184 ceph-mon[102844]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:33 smithi196 ceph-mon[76349]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:35 smithi196 ceph-mon[76349]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:35 smithi184 ceph-mon[100622]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:35 smithi184 ceph-mon[102844]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:37 smithi196 ceph-mon[76349]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:37 smithi184 ceph-mon[100622]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:37 smithi184 ceph-mon[102844]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:37.857 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 5h - - 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 5h - - 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 13s ago 5h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 11s ago 5h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 13s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 11s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:13:38.389 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 13s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 13s ago 5h 519M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 11s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 13s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 13s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 11s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 13s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 13s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 13s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 13s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 11s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 11s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:13:38.390 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 5h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 11s ago 5h 65.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 13s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:13:38.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:38.702 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:13:38.703 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:13:38.969 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:13:38.970 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:13:38.971 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:13:38.971 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:13:39.283 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:13:39.283 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:39.283 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:39.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[100622]: from='client.65853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[100622]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[100622]: from='client.75824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2254372987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[102844]: from='client.65853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[102844]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[102844]: from='client.75824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2254372987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:39 smithi196 ceph-mon[76349]: from='client.65853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:39 smithi196 ceph-mon[76349]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:39 smithi196 ceph-mon[76349]: from='client.75824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2254372987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:40 smithi196 ceph-mon[76349]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:40 smithi196 ceph-mon[76349]: from='client.65877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3046088371' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[102844]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[102844]: from='client.65877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3046088371' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[100622]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[100622]: from='client.65877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3046088371' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:41 smithi196 ceph-mon[76349]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:41.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:41 smithi184 ceph-mon[100622]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:41.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:41 smithi184 ceph-mon[102844]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:41.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:41.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:41.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:13:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:43 smithi196 ceph-mon[76349]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:43 smithi184 ceph-mon[102844]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:43 smithi184 ceph-mon[100622]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:44.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:45 smithi196 ceph-mon[76349]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:45.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:45 smithi184 ceph-mon[100622]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:45.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:45 smithi184 ceph-mon[102844]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[102844]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:46 smithi184 ceph-mon[100622]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:46 smithi196 ceph-mon[76349]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:48.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:13:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:49 smithi196 ceph-mon[76349]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:49.467 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:49 smithi184 ceph-mon[100622]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:49.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:49 smithi184 ceph-mon[102844]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:51.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:51 smithi184 ceph-mon[100622]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:51.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:51 smithi184 ceph-mon[102844]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:51 smithi196 ceph-mon[76349]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:51.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:51.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:13:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:13:51.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:13:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:13:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:13:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:53 smithi184 ceph-mon[100622]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:53 smithi184 ceph-mon[102844]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:53 smithi196 ceph-mon[76349]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:55 smithi184 ceph-mon[100622]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:55 smithi184 ceph-mon[102844]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:55 smithi196 ceph-mon[76349]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:57 smithi184 ceph-mon[100622]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:57 smithi184 ceph-mon[102844]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:57 smithi196 ceph-mon[76349]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:59 smithi184 ceph-mon[100622]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:59 smithi184 ceph-mon[102844]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:59 smithi196 ceph-mon[76349]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:01.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:01 smithi184 ceph-mon[100622]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:01.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:01 smithi184 ceph-mon[102844]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:01 smithi196 ceph-mon[76349]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:01.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:01.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:01.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:02] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:14:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:03 smithi184 ceph-mon[102844]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:03 smithi184 ceph-mon[100622]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:03 smithi196 ceph-mon[76349]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:05 smithi184 ceph-mon[100622]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:05 smithi184 ceph-mon[102844]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:05 smithi196 ceph-mon[76349]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:06 smithi196 ceph-mon[76349]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:14:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[102844]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:14:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:06 smithi184 ceph-mon[100622]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:14:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:08.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:09 smithi184 ceph-mon[100622]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:14:09.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:09 smithi184 ceph-mon[102844]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:14:09.581 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:14:09.587 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:09 smithi196 ceph-mon[76349]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:14:10.119 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:14:10.119 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 5h - - 2024-09-21T01:14:10.119 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 5h - - 2024-09-21T01:14:10.119 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 4s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 4s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 4s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 4s ago 5h 519M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:14:10.120 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 4s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 4s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 4s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 4s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 5h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 5h 87.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:14:10.121 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 5h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 67.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:14:10.122 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:14:10.435 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:14:10.436 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:14:10.437 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:14:10.710 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:14:10.710 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:14:10.710 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:14:10.710 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:14:10.710 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:14:10.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:14:11.027 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:14:11.027 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:11.027 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:11.027 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:11.027 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: from='client.75854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: from='client.75860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1705594723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/865632845' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: from='client.75854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: from='client.75860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1705594723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/865632845' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:11.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:11.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:11.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:11.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:11.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: from='client.75854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: from='client.75860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1705594723' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/865632845' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:12.130 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:12] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T01:14:12.405 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:12 smithi184 ceph-mon[100622]: from='client.75878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:12.406 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:12 smithi184 ceph-mon[102844]: from='client.75878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:12 smithi196 ceph-mon[76349]: from='client.75878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:13.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:13.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:13 smithi196 ceph-mon[76349]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:13 smithi184 ceph-mon[100622]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:13 smithi184 ceph-mon[102844]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:14:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:15 smithi196 ceph-mon[76349]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T01:14:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:15 smithi184 ceph-mon[100622]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T01:14:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:15 smithi184 ceph-mon[102844]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T01:14:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:16 smithi196 ceph-mon[76349]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:16 smithi184 ceph-mon[100622]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:16 smithi184 ceph-mon[102844]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 310 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:19 smithi184 ceph-mon[100622]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T01:14:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:19 smithi184 ceph-mon[102844]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T01:14:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:19 smithi196 ceph-mon[76349]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T01:14:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:21 smithi184 ceph-mon[100622]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:14:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:21 smithi184 ceph-mon[102844]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:14:21.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:21 smithi196 ceph-mon[76349]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:14:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:21.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:21.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:21.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:14:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:23 smithi184 ceph-mon[100622]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:14:23.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:23 smithi184 ceph-mon[102844]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:14:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:23 smithi196 ceph-mon[76349]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 332 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:14:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:25 smithi196 ceph-mon[76349]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:25 smithi184 ceph-mon[100622]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:25 smithi184 ceph-mon[102844]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[100622]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:14:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:26 smithi184 ceph-mon[102844]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:26 smithi196 ceph-mon[76349]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:14:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[100622]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:14:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:28 smithi184 ceph-mon[102844]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:14:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:28 smithi196 ceph-mon[76349]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:14:30.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:31 smithi196 ceph-mon[76349]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:14:31.245 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:31 smithi184 ceph-mon[100622]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:14:31.245 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:31 smithi184 ceph-mon[102844]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:14:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:31.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:31.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:31.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:32.140 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:14:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:33 smithi184 ceph-mon[100622]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:14:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:33 smithi184 ceph-mon[102844]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:14:33.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:33 smithi196 ceph-mon[76349]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:14:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:35 smithi184 ceph-mon[100622]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T01:14:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:35 smithi184 ceph-mon[102844]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T01:14:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:35 smithi196 ceph-mon[76349]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-21T01:14:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:37 smithi184 ceph-mon[100622]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:14:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:37 smithi184 ceph-mon[102844]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:14:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:37 smithi196 ceph-mon[76349]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:14:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:39 smithi184 ceph-mon[100622]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:14:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:39 smithi184 ceph-mon[102844]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:14:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:39 smithi196 ceph-mon[76349]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:14:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:41 smithi184 ceph-mon[100622]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:41 smithi184 ceph-mon[102844]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:41.322 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:14:41.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:41 smithi196 ceph-mon[76349]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:14:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:41.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:41.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:41.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:41.848 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:14:41.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 5h - - 2024-09-21T01:14:41.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 5h - - 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 16s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 14s ago 5h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 16s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 14s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 16s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 16s ago 5h 519M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 14s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 16s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 16s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 14s ago 5h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 16s ago 5h 90.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 16s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:14:41.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 16s ago 5h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 16s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 14s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 14s ago 5h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 14s ago 5h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 5h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 14s ago 5h 68.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 16s ago 5h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:14:41.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:14:42.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:14:42.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:14:42.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:14:42.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:14:42.162 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:42 smithi184 ceph-mon[100622]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:42.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:42 smithi184 ceph-mon[102844]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:42.162 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:14:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:42 smithi196 ceph-mon[76349]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:14:42.428 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:14:42.429 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:14:42.735 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:14:42.736 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:42.736 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:42.736 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:42.736 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: from='client.75902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1329391921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3862439668' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: from='client.75902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1329391921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3862439668' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: from='client.75902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1329391921' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3862439668' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:44 smithi196 ceph-mon[76349]: from='client.65949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:44 smithi184 ceph-mon[100622]: from='client.65949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:44 smithi184 ceph-mon[102844]: from='client.65949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:45 smithi196 ceph-mon[76349]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:14:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:45 smithi184 ceph-mon[100622]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:14:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:45 smithi184 ceph-mon[102844]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:14:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[102844]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T01:14:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:46 smithi184 ceph-mon[100622]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T01:14:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:46 smithi196 ceph-mon[76349]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T01:14:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[100622]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:48 smithi184 ceph-mon[102844]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:14:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:48 smithi196 ceph-mon[76349]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:14:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:51 smithi184 ceph-mon[100622]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:14:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:51 smithi184 ceph-mon[102844]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:14:51.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:51 smithi196 ceph-mon[76349]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:14:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:51.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:51.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:14:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:14:51.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:14:52.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:14:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:14:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:53 smithi184 ceph-mon[100622]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:53 smithi184 ceph-mon[102844]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:53 smithi196 ceph-mon[76349]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:55 smithi196 ceph-mon[76349]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:55 smithi184 ceph-mon[102844]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:55 smithi184 ceph-mon[100622]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:57 smithi196 ceph-mon[76349]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:57 smithi184 ceph-mon[102844]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:57 smithi184 ceph-mon[100622]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:14:59 smithi196 ceph-mon[76349]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:14:59 smithi184 ceph-mon[102844]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:14:59 smithi184 ceph-mon[100622]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:01 smithi196 ceph-mon[76349]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:01.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:01 smithi184 ceph-mon[102844]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:01.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:01 smithi184 ceph-mon[100622]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:01.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:01.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:01.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:15:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:03 smithi196 ceph-mon[76349]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:15:03.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:03.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:03 smithi184 ceph-mon[102844]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:15:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:03 smithi184 ceph-mon[100622]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:15:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:05 smithi196 ceph-mon[76349]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:05 smithi184 ceph-mon[102844]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:05 smithi184 ceph-mon[100622]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[102844]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[100622]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:07 smithi196 ceph-mon[76349]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[100622]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:08 smithi184 ceph-mon[102844]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:08 smithi196 ceph-mon[76349]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:11 smithi196 ceph-mon[76349]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:11.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:11 smithi184 ceph-mon[100622]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:11.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:11 smithi184 ceph-mon[102844]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:11.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:11.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:11.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:12.182 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:15:13.029 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:15:13.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:13.274 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:13 smithi184 ceph-mon[100622]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:13.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:13.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:13 smithi184 ceph-mon[102844]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:13 smithi196 ceph-mon[76349]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 5h 520M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:15:13.579 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 7s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 7s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 7s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 69.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:15:13.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:15:13.581 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:15:13.940 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:15:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:15:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:15:13.942 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:15:14.236 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:15:14.237 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[100622]: from='client.65961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[100622]: from='client.75932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4212012218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[102844]: from='client.65961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[102844]: from='client.75932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4212012218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:14 smithi196 ceph-mon[76349]: from='client.65961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:14 smithi196 ceph-mon[76349]: from='client.75932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4212012218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:14.564 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:15:14.564 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:14.564 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:15 smithi196 ceph-mon[76349]: from='client.75938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:15 smithi196 ceph-mon[76349]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:15 smithi196 ceph-mon[76349]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3093390812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[102844]: from='client.75938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[102844]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[102844]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3093390812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[100622]: from='client.75938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[100622]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[100622]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3093390812' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:17 smithi196 ceph-mon[76349]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:17 smithi184 ceph-mon[100622]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:17 smithi184 ceph-mon[102844]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:19 smithi196 ceph-mon[76349]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:19 smithi184 ceph-mon[100622]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:19 smithi184 ceph-mon[102844]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:21 smithi196 ceph-mon[76349]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:21.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:21 smithi184 ceph-mon[100622]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:21.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:21 smithi184 ceph-mon[102844]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:21.543Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:21.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:21.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:15:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:23 smithi196 ceph-mon[76349]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:23 smithi184 ceph-mon[100622]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:23 smithi184 ceph-mon[102844]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:25 smithi196 ceph-mon[76349]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:25 smithi184 ceph-mon[100622]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:25 smithi184 ceph-mon[102844]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[100622]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[102844]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:27 smithi196 ceph-mon[76349]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[100622]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:28 smithi184 ceph-mon[102844]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:28 smithi196 ceph-mon[76349]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:30 smithi196 ceph-mon[76349]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:30 smithi184 ceph-mon[100622]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:31.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:30 smithi184 ceph-mon[102844]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:31.544Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:31.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:31.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:15:33.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:33.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:33 smithi196 ceph-mon[76349]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:33.493 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:33.493 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:33 smithi184 ceph-mon[100622]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:33.493 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:33.493 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:33 smithi184 ceph-mon[102844]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:35.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:35 smithi184 ceph-mon[100622]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:35.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:35 smithi184 ceph-mon[102844]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:35 smithi196 ceph-mon[76349]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:37 smithi196 ceph-mon[76349]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:37 smithi184 ceph-mon[100622]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:37 smithi184 ceph-mon[102844]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:39 smithi196 ceph-mon[76349]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:39 smithi184 ceph-mon[100622]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:39 smithi184 ceph-mon[102844]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:41 smithi196 ceph-mon[76349]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:41.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:41 smithi184 ceph-mon[102844]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:41.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:41 smithi184 ceph-mon[100622]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:41.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:41.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:41.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:15:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:43 smithi184 ceph-mon[102844]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:43 smithi184 ceph-mon[100622]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:43 smithi196 ceph-mon[76349]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:44.857 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:15:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:45 smithi184 ceph-mon[100622]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:45 smithi184 ceph-mon[102844]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:45.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:45 smithi196 ceph-mon[76349]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 19s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 5h 520M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 19s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:15:45.390 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 17s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 17s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 69.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:15:45.391 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:45.702 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:15:45.703 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:15:45.969 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:15:45.970 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[100622]: from='client.75962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[100622]: from='client.75968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3163112613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[102844]: from='client.75962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[102844]: from='client.75968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3163112613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:46.292 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:15:46.292 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:46.292 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:46 smithi196 ceph-mon[76349]: from='client.75962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:46 smithi196 ceph-mon[76349]: from='client.75968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3163112613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='client.66009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='client.66021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1774599935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='client.66009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='client.66021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1774599935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='client.66009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='client.66021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1774599935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[100622]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:48 smithi184 ceph-mon[102844]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:15:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:48 smithi196 ceph-mon[76349]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:51 smithi196 ceph-mon[76349]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:51.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:51 smithi184 ceph-mon[100622]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:51 smithi184 ceph-mon[102844]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:51.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:51.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:15:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:15:51.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:15:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:15:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:15:53.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:53.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:53 smithi196 ceph-mon[76349]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:53.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:53.455 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:53 smithi184 ceph-mon[100622]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:53.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:53.455 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:53 smithi184 ceph-mon[102844]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:55.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:55 smithi184 ceph-mon[100622]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:55.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:55 smithi184 ceph-mon[102844]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:55 smithi196 ceph-mon[76349]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:57 smithi196 ceph-mon[76349]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:57 smithi184 ceph-mon[100622]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:57 smithi184 ceph-mon[102844]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:59 smithi196 ceph-mon[76349]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:59 smithi184 ceph-mon[100622]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:59.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:59 smithi184 ceph-mon[102844]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:59.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:01 smithi196 ceph-mon[76349]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:01 smithi184 ceph-mon[100622]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:01 smithi184 ceph-mon[102844]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:01.946 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:01.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:01.946 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:01.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:01.946 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:01.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:16:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:03 smithi196 ceph-mon[76349]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:03 smithi184 ceph-mon[100622]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:03 smithi184 ceph-mon[102844]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:05 smithi196 ceph-mon[76349]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:05 smithi184 ceph-mon[100622]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:05 smithi184 ceph-mon[102844]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:07 smithi196 ceph-mon[76349]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[102844]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:07.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[100622]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[102844]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[100622]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:09 smithi196 ceph-mon[76349]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:11 smithi184 ceph-mon[100622]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:11 smithi184 ceph-mon[102844]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:11 smithi196 ceph-mon[76349]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:11.957 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:11.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:11.957 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:11.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:11.957 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:11.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:16:13.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:13.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:13 smithi196 ceph-mon[76349]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:13 smithi184 ceph-mon[102844]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:13 smithi184 ceph-mon[100622]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:15 smithi196 ceph-mon[76349]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:15 smithi184 ceph-mon[100622]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:15 smithi184 ceph-mon[102844]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:16.581 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 5h - - 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 10s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 10s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:16:17.112 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 10s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 10s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 10s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 10s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 10s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 10s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 10s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 9s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:16:17.113 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 9s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:16:17.114 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 5h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:16:17.114 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:16:17.114 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:16:17.114 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:16:17.114 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 9s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:16:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:17 smithi184 ceph-mon[100622]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:17 smithi184 ceph-mon[102844]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:17 smithi196 ceph-mon[76349]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:16:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:17.426 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:16:17.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:16:17.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:16:17.426 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:16:17.696 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:16:17.697 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:16:17.697 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:16:17.697 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:16:17.697 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:16:18.000 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:16:18.001 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:18.001 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[100622]: from='client.66033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[100622]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[100622]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/42080680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/669059739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[102844]: from='client.66033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[102844]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[102844]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/42080680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/669059739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:18 smithi196 ceph-mon[76349]: from='client.66033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:18 smithi196 ceph-mon[76349]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:18 smithi196 ceph-mon[76349]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/42080680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/669059739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:19 smithi196 ceph-mon[76349]: from='client.76022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:19 smithi196 ceph-mon[76349]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:19 smithi184 ceph-mon[100622]: from='client.76022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:19 smithi184 ceph-mon[100622]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:19 smithi184 ceph-mon[102844]: from='client.76022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:19 smithi184 ceph-mon[102844]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:21 smithi196 ceph-mon[76349]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:21 smithi184 ceph-mon[100622]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:21 smithi184 ceph-mon[102844]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:21.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:21.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:21.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:21.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:21.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:21.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:16:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:23 smithi196 ceph-mon[76349]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:23 smithi184 ceph-mon[102844]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:23 smithi184 ceph-mon[100622]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:25 smithi196 ceph-mon[76349]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:25 smithi184 ceph-mon[100622]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:25 smithi184 ceph-mon[102844]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:27 smithi196 ceph-mon[76349]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[100622]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[102844]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[100622]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[102844]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:29 smithi196 ceph-mon[76349]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:31 smithi184 ceph-mon[100622]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:31 smithi184 ceph-mon[102844]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:31 smithi196 ceph-mon[76349]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:31.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:31.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:31.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:31.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:31.978 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:31.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:16:33.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:33.475 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:33 smithi184 ceph-mon[100622]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:16:33.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:33.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:33 smithi184 ceph-mon[102844]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:16:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:33 smithi196 ceph-mon[76349]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:16:35.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:35 smithi184 ceph-mon[102844]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:35.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:35 smithi184 ceph-mon[100622]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:35 smithi196 ceph-mon[76349]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:37 smithi184 ceph-mon[100622]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:37 smithi184 ceph-mon[102844]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:37 smithi196 ceph-mon[76349]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:39 smithi184 ceph-mon[100622]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:39 smithi184 ceph-mon[102844]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:39 smithi196 ceph-mon[76349]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:41 smithi184 ceph-mon[100622]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:41 smithi184 ceph-mon[102844]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:41 smithi196 ceph-mon[76349]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:41.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:41.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:41.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:16:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:43 smithi184 ceph-mon[100622]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:43 smithi184 ceph-mon[102844]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:43 smithi196 ceph-mon[76349]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:44.452 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:45 smithi184 ceph-mon[100622]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:45 smithi184 ceph-mon[102844]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:45 smithi196 ceph-mon[76349]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[100622]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[102844]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:47 smithi196 ceph-mon[76349]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:48.289 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 5h - - 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 0s ago 5h 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 0s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:16:48.827 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 0s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 0s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 0s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:16:48.828 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 0s ago 5h 67.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:16:48.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:16:48.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:16:48.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:16:48.829 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 0s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:16:49.133 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:16:49.133 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:16:49.133 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:16:49.134 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:16:49.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:16:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:16:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:16:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:16:49.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:16:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:49.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: from='client.66069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2963365117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: from='client.66069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2963365117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: from='client.66069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2963365117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:16:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:49.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:49.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:50 smithi196 ceph-mon[76349]: from='client.76040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:50 smithi196 ceph-mon[76349]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2243590381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[100622]: from='client.76040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[100622]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2243590381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[102844]: from='client.76040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[102844]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2243590381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:51 smithi196 ceph-mon[76349]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:51 smithi196 ceph-mon[76349]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-mon[100622]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-mon[100622]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-mon[102844]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-mon[102844]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:51.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:51.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:16:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:16:51.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:16:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:16:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:16:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:53 smithi196 ceph-mon[76349]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:53 smithi184 ceph-mon[100622]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:53 smithi184 ceph-mon[102844]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:55 smithi196 ceph-mon[76349]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:55.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:55 smithi184 ceph-mon[100622]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:55.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:55 smithi184 ceph-mon[102844]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:57 smithi184 ceph-mon[100622]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:57 smithi184 ceph-mon[102844]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:57 smithi196 ceph-mon[76349]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:59 smithi196 ceph-mon[76349]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:59 smithi184 ceph-mon[100622]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:59 smithi184 ceph-mon[102844]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:16:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:01 smithi196 ceph-mon[76349]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:01 smithi184 ceph-mon[102844]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:01.963 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:01 smithi184 ceph-mon[100622]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:01.963 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:01.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:01.964 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:01.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:01.964 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:01.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:17:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:03 smithi196 ceph-mon[76349]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:03 smithi184 ceph-mon[102844]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:03 smithi184 ceph-mon[100622]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:05 smithi196 ceph-mon[76349]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:05 smithi184 ceph-mon[100622]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:05 smithi184 ceph-mon[102844]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:07 smithi196 ceph-mon[76349]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[100622]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[102844]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:09 smithi196 ceph-mon[76349]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[100622]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[102844]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:11 smithi196 ceph-mon[76349]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:11 smithi184 ceph-mon[100622]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:11 smithi184 ceph-mon[102844]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:11.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:11.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:11.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:17:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:12 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:12 smithi184 ceph-mon[100622]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:12 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:12 smithi184 ceph-mon[102844]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:12 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:12 smithi196 ceph-mon[76349]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:15 smithi196 ceph-mon[76349]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:15 smithi184 ceph-mon[102844]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:15 smithi184 ceph-mon[100622]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:17 smithi196 ceph-mon[76349]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:17.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:17 smithi184 ceph-mon[100622]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:17.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:17 smithi184 ceph-mon[102844]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:19.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:19 smithi184 ceph-mon[100622]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:19.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:19 smithi184 ceph-mon[102844]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:19 smithi196 ceph-mon[76349]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:19.997 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 5h - - 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 13s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 12s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 13s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:17:20.524 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 13s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 13s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 13s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 13s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 12s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 13s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 13s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 13s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:17:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 13s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 12s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 5h 67.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:17:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 12s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:17:20.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:17:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:17:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:17:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:17:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:17:20.837 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:17:21.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:17:21.104 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:17:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:21 smithi196 ceph-mon[76349]: from='client.66105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:21 smithi196 ceph-mon[76349]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:21 smithi196 ceph-mon[76349]: from='client.76076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3044682948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:21.414 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:17:21.414 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:21.414 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:21.414 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:21.414 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:21.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[100622]: from='client.66105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[100622]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[100622]: from='client.76076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3044682948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[102844]: from='client.66105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[102844]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[102844]: from='client.76076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:21.415 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3044682948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:21.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:21.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:21.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[100622]: from='client.76082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[100622]: from='client.66129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2261373655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[102844]: from='client.76082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[102844]: from='client.66129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2261373655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:22 smithi196 ceph-mon[76349]: from='client.76082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:22 smithi196 ceph-mon[76349]: from='client.66129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2261373655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:23.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:23 smithi196 ceph-mon[76349]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:23 smithi184 ceph-mon[100622]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:23 smithi184 ceph-mon[102844]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:25 smithi196 ceph-mon[76349]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:25 smithi184 ceph-mon[100622]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:25 smithi184 ceph-mon[102844]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[100622]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[102844]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:27 smithi196 ceph-mon[76349]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[102844]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[100622]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:29 smithi196 ceph-mon[76349]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:31 smithi184 ceph-mon[100622]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:31 smithi184 ceph-mon[102844]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:31 smithi196 ceph-mon[76349]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:31.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:31.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:31.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:17:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:33 smithi184 ceph-mon[100622]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:33 smithi184 ceph-mon[102844]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:33 smithi196 ceph-mon[76349]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:35 smithi184 ceph-mon[100622]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:35 smithi184 ceph-mon[102844]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:35 smithi196 ceph-mon[76349]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:37 smithi196 ceph-mon[76349]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:37.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:37 smithi184 ceph-mon[100622]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:37.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:37 smithi184 ceph-mon[102844]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:39.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:39 smithi184 ceph-mon[100622]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:39.506 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:39 smithi184 ceph-mon[102844]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:39 smithi196 ceph-mon[76349]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:41 smithi184 ceph-mon[100622]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:41 smithi184 ceph-mon[102844]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:41 smithi196 ceph-mon[76349]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:41.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:41.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:41.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:17:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:43 smithi184 ceph-mon[100622]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:43 smithi184 ceph-mon[102844]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:43 smithi196 ceph-mon[76349]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:45 smithi196 ceph-mon[76349]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:45 smithi184 ceph-mon[100622]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:45 smithi184 ceph-mon[102844]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[100622]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[102844]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:47 smithi196 ceph-mon[76349]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[100622]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[102844]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:49 smithi196 ceph-mon[76349]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:17:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:51 smithi184 ceph-mon[100622]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:51 smithi184 ceph-mon[102844]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:51 smithi196 ceph-mon[76349]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:17:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:51.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:52.241 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:51.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:17:52.242 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:17:51.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 3s ago 5h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:17:52.245 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 5h 522M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 5s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 3s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 3s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 5h 68.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:17:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:17:52.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:17:52.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:17:52.247 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:17:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:17:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:17:52.557 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:17:52.557 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:17:52.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:17:52.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:17:52.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:17:52.559 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:17:52.826 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:17:52.827 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:17:53.135 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:17:53.135 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:53.135 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:53.135 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:53.135 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.66141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.66147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1212549788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1609089945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.66141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.66147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1212549788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:53.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1609089945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.66141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.66147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1212549788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1609089945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:54 smithi184 ceph-mon[100622]: from='client.66165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:54 smithi184 ceph-mon[102844]: from='client.66165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:54 smithi196 ceph-mon[76349]: from='client.66165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:55 smithi184 ceph-mon[100622]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:55 smithi184 ceph-mon[102844]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:55 smithi196 ceph-mon[76349]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:57.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:57 smithi196 ceph-mon[76349]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:57 smithi184 ceph-mon[102844]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:57 smithi184 ceph-mon[100622]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:59.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:59 smithi184 ceph-mon[100622]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:59.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:59.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:59 smithi184 ceph-mon[102844]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:59.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:59.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:59 smithi196 ceph-mon[76349]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:59.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:01 smithi184 ceph-mon[100622]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:01 smithi184 ceph-mon[102844]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:01 smithi196 ceph-mon[76349]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:01.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:01.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:01.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:18:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:03 smithi184 ceph-mon[100622]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:03 smithi184 ceph-mon[102844]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:03 smithi196 ceph-mon[76349]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:05 smithi184 ceph-mon[100622]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:05 smithi184 ceph-mon[102844]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:05 smithi196 ceph-mon[76349]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[100622]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[102844]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:07 smithi196 ceph-mon[76349]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[100622]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[102844]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:09 smithi196 ceph-mon[76349]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:11 smithi184 ceph-mon[100622]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:11.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:11 smithi184 ceph-mon[102844]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:11 smithi196 ceph-mon[76349]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:11.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:11.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:11.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:18:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:13 smithi184 ceph-mon[102844]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:13 smithi184 ceph-mon[100622]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:13 smithi196 ceph-mon[76349]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:18:13.812058066Z level=info msg="Completed cleanup jobs" duration=12.587128ms 2024-09-21T01:18:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:15 smithi184 ceph-mon[100622]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:15 smithi184 ceph-mon[102844]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:15 smithi196 ceph-mon[76349]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:17.333 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:17 smithi196 ceph-mon[76349]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:17 smithi184 ceph-mon[100622]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:17 smithi184 ceph-mon[102844]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:19 smithi184 ceph-mon[100622]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:19 smithi184 ceph-mon[102844]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:19 smithi196 ceph-mon[76349]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:21 smithi184 ceph-mon[102844]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:21 smithi184 ceph-mon[100622]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:21 smithi196 ceph-mon[76349]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:21.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:21.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:21.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:18:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:18:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:23 smithi184 ceph-mon[100622]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:23 smithi184 ceph-mon[102844]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:23 smithi196 ceph-mon[76349]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 5h - - 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 16s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 15s ago 5h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 16s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 16s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:18:23.967 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 16s ago 5h 522M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 16s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 16s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 15s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 16s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 16s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 16s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 16s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 5h 66.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:18:23.968 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:18:23.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:18:23.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:18:23.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:24.275 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:18:24.276 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:18:24.547 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:18:24.548 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:18:24.856 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:18:24.856 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:24.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:24.856 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:24.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: from='client.66189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1349091596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2435828535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: from='client.66189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1349091596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2435828535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: from='client.66189 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1349091596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2435828535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:26 smithi184 ceph-mon[100622]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:26 smithi184 ceph-mon[102844]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:26 smithi196 ceph-mon[76349]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:27 smithi184 ceph-mon[100622]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:27 smithi184 ceph-mon[102844]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:27 smithi196 ceph-mon[76349]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:29 smithi196 ceph-mon[76349]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[100622]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[102844]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:31 smithi196 ceph-mon[76349]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:31 smithi184 ceph-mon[100622]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:31 smithi184 ceph-mon[102844]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:31.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:31.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:31.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:18:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:33 smithi196 ceph-mon[76349]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:33.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:33 smithi184 ceph-mon[100622]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:33 smithi184 ceph-mon[102844]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:35 smithi196 ceph-mon[76349]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:35 smithi184 ceph-mon[100622]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:35 smithi184 ceph-mon[102844]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:37 smithi196 ceph-mon[76349]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:37 smithi184 ceph-mon[100622]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:37 smithi184 ceph-mon[102844]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:39.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:39 smithi196 ceph-mon[76349]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:39.487 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:39 smithi184 ceph-mon[100622]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:39.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:39 smithi184 ceph-mon[102844]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:41.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:41 smithi184 ceph-mon[100622]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:41.420 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:41 smithi184 ceph-mon[102844]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:41 smithi196 ceph-mon[76349]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:41.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:41.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:41.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:18:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:43 smithi196 ceph-mon[76349]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:43 smithi184 ceph-mon[102844]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:43 smithi184 ceph-mon[100622]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:45 smithi196 ceph-mon[76349]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:45 smithi184 ceph-mon[100622]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:45 smithi184 ceph-mon[102844]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:47 smithi196 ceph-mon[76349]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:47 smithi184 ceph-mon[100622]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:47 smithi184 ceph-mon[102844]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[100622]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:48 smithi184 ceph-mon[102844]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:48 smithi196 ceph-mon[76349]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:18:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:51 smithi196 ceph-mon[76349]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:51 smithi184 ceph-mon[100622]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:51 smithi184 ceph-mon[102844]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:51.870 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:51.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:51.871 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:51.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:18:51.871 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:18:51.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:18:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:18:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:53 smithi196 ceph-mon[76349]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:53 smithi184 ceph-mon[100622]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:53 smithi184 ceph-mon[102844]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:55.155 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:18:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:55 smithi196 ceph-mon[76349]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:55 smithi184 ceph-mon[100622]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:55 smithi184 ceph-mon[102844]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 8s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 6s ago 5h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:18:55.683 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 5h 523M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 8s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 6s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 6s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 6s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:18:55.684 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 6s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:18:55.685 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 5h 68.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:18:55.685 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:18:55.685 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:18:55.685 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:18:55.685 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:18:55.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:18:55.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:18:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:18:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:18:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:18:56.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:18:56.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:56 smithi184 ceph-mon[100622]: from='client.66213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:56.269 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/673301750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:56.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:56 smithi184 ceph-mon[102844]: from='client.66213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:56.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/673301750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:56 smithi196 ceph-mon[76349]: from='client.66213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/673301750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:56.579 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:18:56.579 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:56.579 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[102844]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[102844]: from='client.76190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[102844]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[102844]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2541406576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[100622]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[100622]: from='client.76190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[100622]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[100622]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2541406576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:57 smithi196 ceph-mon[76349]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:57 smithi196 ceph-mon[76349]: from='client.76190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:57 smithi196 ceph-mon[76349]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:57 smithi196 ceph-mon[76349]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2541406576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:59.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:59 smithi196 ceph-mon[76349]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:59.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:59 smithi184 ceph-mon[100622]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:59 smithi184 ceph-mon[102844]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:01.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:01 smithi184 ceph-mon[100622]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:01.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:01 smithi184 ceph-mon[102844]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:01 smithi196 ceph-mon[76349]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:01.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:01.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:01.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:02.181 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:19:03.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:03 smithi184 ceph-mon[100622]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:03.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:03.501 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:03 smithi184 ceph-mon[102844]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:03.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:03 smithi196 ceph-mon[76349]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:05 smithi184 ceph-mon[100622]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:05 smithi184 ceph-mon[102844]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:05 smithi196 ceph-mon[76349]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:07 smithi184 ceph-mon[100622]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:07.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:07 smithi184 ceph-mon[102844]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:07 smithi196 ceph-mon[76349]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:08 smithi196 ceph-mon[76349]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[100622]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:08 smithi184 ceph-mon[102844]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:11 smithi184 ceph-mon[100622]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:11.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:11 smithi184 ceph-mon[102844]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:11 smithi196 ceph-mon[76349]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:11.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:11.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:11.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:19:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:13 smithi184 ceph-mon[100622]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:13 smithi184 ceph-mon[102844]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:13 smithi196 ceph-mon[76349]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:14.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:15 smithi196 ceph-mon[76349]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:15 smithi184 ceph-mon[100622]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:15 smithi184 ceph-mon[102844]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:17 smithi184 ceph-mon[100622]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:17 smithi184 ceph-mon[102844]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:17 smithi196 ceph-mon[76349]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:19.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:19 smithi196 ceph-mon[76349]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:19 smithi184 ceph-mon[100622]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:19 smithi184 ceph-mon[102844]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:21 smithi196 ceph-mon[76349]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:21 smithi184 ceph-mon[100622]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:21 smithi184 ceph-mon[102844]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:21.829 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:21.561Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:21.829 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:21.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:21.829 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:21.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:22.128 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:19:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:23 smithi184 ceph-mon[100622]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:23.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:23 smithi184 ceph-mon[102844]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:23.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:23 smithi196 ceph-mon[76349]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:25 smithi184 ceph-mon[100622]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:25 smithi184 ceph-mon[102844]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:25 smithi196 ceph-mon[76349]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:26.876 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 19s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 5h 523M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:19:27.399 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 19s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 18s ago 5h 18.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 5h 68.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:19:27.400 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:19:27.401 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:19:27.401 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:19:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:27 smithi184 ceph-mon[100622]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:27 smithi184 ceph-mon[102844]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:27 smithi196 ceph-mon[76349]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:27.740 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:19:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:19:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:19:27.741 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:19:28.009 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:19:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:19:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:19:28.010 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='client.76214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/245047812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='client.76214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/245047812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:28.319 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:19:28.319 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:28.319 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:28.319 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:28.319 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='client.76214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/245047812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[100622]: from='client.76226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[100622]: from='client.66273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[100622]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3026380234' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[102844]: from='client.76226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[102844]: from='client.66273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[102844]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3026380234' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:29 smithi196 ceph-mon[76349]: from='client.76226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:29 smithi196 ceph-mon[76349]: from='client.66273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:29 smithi196 ceph-mon[76349]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3026380234' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:31 smithi196 ceph-mon[76349]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:31 smithi184 ceph-mon[100622]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:31 smithi184 ceph-mon[102844]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:31.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:31.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:31.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:19:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:33 smithi184 ceph-mon[100622]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:33 smithi184 ceph-mon[102844]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:33 smithi196 ceph-mon[76349]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:35 smithi196 ceph-mon[76349]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:35 smithi184 ceph-mon[100622]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:35 smithi184 ceph-mon[102844]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:37 smithi196 ceph-mon[76349]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:37 smithi184 ceph-mon[100622]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:37 smithi184 ceph-mon[102844]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:39 smithi196 ceph-mon[76349]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:39 smithi184 ceph-mon[100622]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:39 smithi184 ceph-mon[102844]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:41.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:41 smithi196 ceph-mon[76349]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:41.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:41 smithi184 ceph-mon[100622]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:41.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:41 smithi184 ceph-mon[102844]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:41.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:41.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:41.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:41.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:41.628 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:41.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:19:43.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:43 smithi184 ceph-mon[100622]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:43.404 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:43 smithi184 ceph-mon[102844]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:43.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:43 smithi196 ceph-mon[76349]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:45 smithi196 ceph-mon[76349]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:45 smithi184 ceph-mon[100622]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:45 smithi184 ceph-mon[102844]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:47 smithi196 ceph-mon[76349]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:47 smithi184 ceph-mon[100622]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:47 smithi184 ceph-mon[102844]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:48 smithi196 ceph-mon[76349]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[100622]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:48 smithi184 ceph-mon[102844]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[100622]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:50 smithi184 ceph-mon[102844]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:19:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:50 smithi196 ceph-mon[76349]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:51.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:51.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:19:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:19:51.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:19:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:19:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:19:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:53 smithi184 ceph-mon[100622]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:53 smithi184 ceph-mon[102844]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:53 smithi196 ceph-mon[76349]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:55 smithi184 ceph-mon[100622]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:55 smithi184 ceph-mon[102844]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:55 smithi196 ceph-mon[76349]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:57 smithi184 ceph-mon[100622]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:57 smithi184 ceph-mon[102844]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:57 smithi196 ceph-mon[76349]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:19:59.139 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:19:59.139 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 11s ago 5h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 11s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 11s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 11s ago 5h 524M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 11s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 11s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:19:59.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 11s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 11s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 11s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 9s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 9s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 5h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:19:59.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 9s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:19:59.450 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:19:59.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:19:59.451 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:19:59.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:19:59.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:19:59.451 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:19:59.451 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:59 smithi184 ceph-mon[100622]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:59.451 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:59.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:59 smithi184 ceph-mon[102844]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:59.451 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:59 smithi196 ceph-mon[76349]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:59.722 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:19:59.722 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:19:59.722 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:19:59.722 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:19:59.722 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:19:59.723 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:20:00.035 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.036 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:00.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:00.036 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.036 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: from='client.66285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: from='client.76256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2740742124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3283989075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:19:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: from='client.66285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: from='client.76256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2740742124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3283989075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:00.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: from='client.66285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: from='client.76256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:00.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2740742124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:00.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3283989075' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:01.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:01 smithi196 ceph-mon[76349]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:01.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:01 smithi196 ceph-mon[76349]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-mon[100622]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-mon[100622]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-mon[102844]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-mon[102844]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:01.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:01.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:01.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:20:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:03 smithi184 ceph-mon[100622]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:03 smithi184 ceph-mon[102844]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:03 smithi196 ceph-mon[76349]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:05 smithi184 ceph-mon[100622]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:05 smithi184 ceph-mon[102844]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:05 smithi196 ceph-mon[76349]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:07 smithi184 ceph-mon[100622]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:07.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:07 smithi184 ceph-mon[102844]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:07 smithi196 ceph-mon[76349]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[100622]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:08 smithi184 ceph-mon[102844]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:08 smithi196 ceph-mon[76349]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:10 smithi196 ceph-mon[76349]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[100622]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:10 smithi184 ceph-mon[102844]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:11.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:11.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:11.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:20:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:13 smithi184 ceph-mon[102844]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:13 smithi184 ceph-mon[100622]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:13 smithi196 ceph-mon[76349]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:14.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:15 smithi184 ceph-mon[100622]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:15 smithi184 ceph-mon[102844]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:15 smithi196 ceph-mon[76349]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:17 smithi184 ceph-mon[100622]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:17 smithi184 ceph-mon[102844]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:17 smithi196 ceph-mon[76349]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:19 smithi184 ceph-mon[100622]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:19 smithi184 ceph-mon[102844]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:19 smithi196 ceph-mon[76349]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:21 smithi184 ceph-mon[100622]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:21 smithi184 ceph-mon[102844]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:21.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:21 smithi196 ceph-mon[76349]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:21.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:21.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:21.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:20:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:23 smithi196 ceph-mon[76349]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:23.392 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:23 smithi184 ceph-mon[100622]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:23.393 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:23.393 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:23 smithi184 ceph-mon[102844]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:23.393 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:25.471 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:25 smithi184 ceph-mon[100622]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:25.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:25 smithi184 ceph-mon[102844]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:25 smithi196 ceph-mon[76349]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:27 smithi184 ceph-mon[100622]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:27 smithi184 ceph-mon[102844]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:27 smithi196 ceph-mon[76349]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:29 smithi196 ceph-mon[76349]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[100622]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[102844]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:30.335 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 5h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:20:30.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 18.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 1s ago 5h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:20:30.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:20:30.862 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[100622]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[100622]: from='client.66321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[102844]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:30 smithi184 ceph-mon[102844]: from='client.66321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:30 smithi196 ceph-mon[76349]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:30 smithi196 ceph-mon[76349]: from='client.66321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:31.174 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:20:31.174 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:20:31.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:20:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:20:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:20:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:20:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:20:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:20:31.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:20:31.723 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:31.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:31.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:31.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:31.724 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:31.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:31.759 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:31.759 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:31.759 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:32.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[102844]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[102844]: from='client.76298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2044195489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[100622]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[100622]: from='client.76298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2044195489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:31 smithi196 ceph-mon[76349]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:31 smithi196 ceph-mon[76349]: from='client.76298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2044195489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:20:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[100622]: from='client.66345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3324629975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[100622]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[102844]: from='client.66345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3324629975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[102844]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:32 smithi196 ceph-mon[76349]: from='client.66345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3324629975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:32 smithi196 ceph-mon[76349]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:35 smithi184 ceph-mon[100622]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:35 smithi184 ceph-mon[102844]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:35 smithi196 ceph-mon[76349]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:37 smithi184 ceph-mon[102844]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:37 smithi184 ceph-mon[100622]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:37 smithi196 ceph-mon[76349]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:39 smithi184 ceph-mon[100622]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:39 smithi184 ceph-mon[102844]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:39.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:39 smithi196 ceph-mon[76349]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:41.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:41 smithi196 ceph-mon[76349]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:41 smithi184 ceph-mon[100622]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:41 smithi184 ceph-mon[102844]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:41.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:41.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:41.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:20:43.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:43 smithi196 ceph-mon[76349]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:43.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:43 smithi184 ceph-mon[100622]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:43 smithi184 ceph-mon[102844]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:45.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:45 smithi184 ceph-mon[100622]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:45.256 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:45 smithi184 ceph-mon[102844]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:45 smithi196 ceph-mon[76349]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:47 smithi184 ceph-mon[100622]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:47 smithi184 ceph-mon[102844]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:47 smithi196 ceph-mon[76349]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[100622]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[102844]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:49 smithi196 ceph-mon[76349]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:50 smithi196 ceph-mon[76349]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[100622]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:20:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:50 smithi184 ceph-mon[102844]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:51.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:51.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:20:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:20:51.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:20:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:20:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:20:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:53 smithi184 ceph-mon[100622]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:53 smithi184 ceph-mon[102844]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:53.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:53 smithi196 ceph-mon[76349]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:53.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:55 smithi184 ceph-mon[100622]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:55 smithi184 ceph-mon[102844]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:55 smithi196 ceph-mon[76349]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:57 smithi184 ceph-mon[100622]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:57 smithi184 ceph-mon[102844]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:57 smithi196 ceph-mon[76349]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:59 smithi184 ceph-mon[100622]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:59 smithi184 ceph-mon[102844]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:59 smithi196 ceph-mon[76349]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:01 smithi184 ceph-mon[100622]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:01 smithi184 ceph-mon[102844]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:01 smithi196 ceph-mon[76349]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:01.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:01.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:01.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:02.062 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:21:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:21:02.615 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 12s ago 5h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:21:02.616 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 12s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 12s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 5h 70.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 12s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:21:02.938 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:21:02.938 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:21:02.938 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:21:02.939 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:21:03.218 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:21:03.219 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:21:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:03 smithi196 ceph-mon[76349]: from='client.66357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:03 smithi196 ceph-mon[76349]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:03 smithi196 ceph-mon[76349]: from='client.76328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/722767078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:03.526 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:21:03.527 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:03.527 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:03.527 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:03.527 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[100622]: from='client.66357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[100622]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[100622]: from='client.76328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/722767078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:03.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[102844]: from='client.66357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[102844]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:03.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:03.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[102844]: from='client.76328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:03.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/722767078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:04 smithi196 ceph-mon[76349]: from='client.76334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:04 smithi196 ceph-mon[76349]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2535343852' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:04.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[100622]: from='client.76334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[100622]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2535343852' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:04.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[102844]: from='client.76334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[102844]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:04.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2535343852' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:05 smithi184 ceph-mon[100622]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:05 smithi184 ceph-mon[102844]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:05 smithi196 ceph-mon[76349]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:07 smithi184 ceph-mon[100622]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:07 smithi184 ceph-mon[102844]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:07 smithi196 ceph-mon[76349]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:09 smithi196 ceph-mon[76349]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[100622]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[102844]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[100622]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:10 smithi184 ceph-mon[102844]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:10 smithi196 ceph-mon[76349]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:11.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:11.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:11.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:21:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:13 smithi184 ceph-mon[100622]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:13 smithi184 ceph-mon[102844]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:13 smithi196 ceph-mon[76349]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:14.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:15 smithi184 ceph-mon[100622]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:15 smithi184 ceph-mon[102844]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:15.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:15 smithi196 ceph-mon[76349]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:17 smithi184 ceph-mon[100622]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:17 smithi184 ceph-mon[102844]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:17 smithi196 ceph-mon[76349]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:19 smithi184 ceph-mon[100622]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:19 smithi184 ceph-mon[102844]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:19 smithi196 ceph-mon[76349]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:21 smithi184 ceph-mon[100622]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:21 smithi184 ceph-mon[102844]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:21 smithi196 ceph-mon[76349]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:21.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:21.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:21.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:21:23.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:23 smithi196 ceph-mon[76349]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:23.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:23 smithi184 ceph-mon[100622]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:23 smithi184 ceph-mon[102844]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:25.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:25 smithi184 ceph-mon[100622]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:25.442 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:25 smithi184 ceph-mon[102844]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:25 smithi196 ceph-mon[76349]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:27.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:27 smithi184 ceph-mon[100622]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:27.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:27 smithi184 ceph-mon[102844]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:27 smithi196 ceph-mon[76349]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[100622]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[102844]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:29 smithi196 ceph-mon[76349]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:31 smithi196 ceph-mon[76349]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[100622]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-mon[102844]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:31.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:31.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:31.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:21:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:33 smithi184 ceph-mon[100622]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:33 smithi184 ceph-mon[102844]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:33 smithi196 ceph-mon[76349]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:33.824 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 5h - - 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 5h 29.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 4s ago 5h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 4s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 4s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:21:34.355 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 5s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 4s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 4s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 4s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 4s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 4s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 4s ago 5h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:21:34.356 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:21:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 4s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:21:34.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:21:34.663 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:21:34.930 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:21:34.931 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:21:35.252 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:21:35.252 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:35.252 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:35.252 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[102844]: from='client.76358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[102844]: from='client.76364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[102844]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[102844]: from='client.76370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/685579461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[100622]: from='client.76358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[100622]: from='client.76364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[100622]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[100622]: from='client.76370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/685579461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:35 smithi196 ceph-mon[76349]: from='client.76358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:35 smithi196 ceph-mon[76349]: from='client.76364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:35 smithi196 ceph-mon[76349]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:35 smithi196 ceph-mon[76349]: from='client.76370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/685579461' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:36 smithi184 ceph-mon[100622]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2337906084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:36 smithi184 ceph-mon[102844]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2337906084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:36 smithi196 ceph-mon[76349]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2337906084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:37 smithi184 ceph-mon[100622]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:37 smithi184 ceph-mon[102844]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:37 smithi196 ceph-mon[76349]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:39 smithi184 ceph-mon[100622]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:39 smithi184 ceph-mon[102844]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:39 smithi196 ceph-mon[76349]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:41 smithi184 ceph-mon[102844]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:41 smithi184 ceph-mon[100622]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:41 smithi196 ceph-mon[76349]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:41.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:41.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:41.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:42.260 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:21:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:43 smithi184 ceph-mon[100622]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:43 smithi184 ceph-mon[102844]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:43.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:43 smithi196 ceph-mon[76349]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:43.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:44.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:45.500 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:45 smithi196 ceph-mon[76349]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:45 smithi184 ceph-mon[100622]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:45 smithi184 ceph-mon[102844]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:47.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:47 smithi196 ceph-mon[76349]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:47.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:47 smithi184 ceph-mon[100622]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:47.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:47 smithi184 ceph-mon[102844]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[100622]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[102844]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:49 smithi196 ceph-mon[76349]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[100622]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[102844]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:51 smithi196 ceph-mon[76349]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:51.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:51.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:21:51.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:21:52.270 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:21:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:21:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:53 smithi184 ceph-mon[100622]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:53 smithi184 ceph-mon[102844]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:53 smithi196 ceph-mon[76349]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:55 smithi184 ceph-mon[100622]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:55 smithi184 ceph-mon[102844]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:55 smithi196 ceph-mon[76349]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:57 smithi196 ceph-mon[76349]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:57 smithi184 ceph-mon[100622]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:57 smithi184 ceph-mon[102844]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:59 smithi184 ceph-mon[100622]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:59 smithi184 ceph-mon[102844]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:59 smithi196 ceph-mon[76349]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:01 smithi184 ceph-mon[100622]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:01 smithi184 ceph-mon[102844]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:01 smithi196 ceph-mon[76349]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:01.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:01.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:01.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:02.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:22:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:03 smithi184 ceph-mon[100622]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:03 smithi184 ceph-mon[102844]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:03.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:03 smithi196 ceph-mon[76349]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:03.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:05 smithi196 ceph-mon[76349]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:05.545 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:05 smithi184 ceph-mon[100622]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:05.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:05 smithi184 ceph-mon[102844]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:05.545 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:22:06.080 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:22:06.080 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T01:22:06.080 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T01:22:06.080 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:22:06.080 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 15s ago 5h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 15s ago 5h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:22:06.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 5h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:22:06.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:22:06.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:22:06.393 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:22:06.661 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:22:06.979 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:22:06.979 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:06.979 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:06.979 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:06.979 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: from='client.66429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: from='client.76400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1415244646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/713271799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: from='client.66429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: from='client.76400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1415244646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/713271799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: from='client.66429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: from='client.76400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1415244646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/713271799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:08 smithi184 ceph-mon[100622]: from='client.66453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:08 smithi184 ceph-mon[102844]: from='client.66453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:08 smithi196 ceph-mon[76349]: from='client.66453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[100622]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[102844]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:09 smithi196 ceph-mon[76349]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:11 smithi196 ceph-mon[76349]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[100622]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[102844]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:11.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:11.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:11.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:12.284 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:22:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:13 smithi196 ceph-mon[76349]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:13 smithi184 ceph-mon[100622]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:13 smithi184 ceph-mon[102844]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:15 smithi196 ceph-mon[76349]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:15 smithi184 ceph-mon[100622]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:15 smithi184 ceph-mon[102844]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:17 smithi196 ceph-mon[76349]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:17 smithi184 ceph-mon[100622]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:17 smithi184 ceph-mon[102844]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:19 smithi196 ceph-mon[76349]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:19 smithi184 ceph-mon[100622]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:19 smithi184 ceph-mon[102844]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:21 smithi196 ceph-mon[76349]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:21 smithi184 ceph-mon[100622]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:21 smithi184 ceph-mon[102844]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:21.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:21.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:21.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:22.293 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:22:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:23 smithi196 ceph-mon[76349]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:23 smithi184 ceph-mon[100622]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:23 smithi184 ceph-mon[102844]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:25 smithi196 ceph-mon[76349]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:25 smithi184 ceph-mon[100622]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:25 smithi184 ceph-mon[102844]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:27 smithi196 ceph-mon[76349]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:27.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:27 smithi184 ceph-mon[100622]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:27.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:27 smithi184 ceph-mon[102844]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:29.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[100622]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:29.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:29.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[102844]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:29.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:29 smithi196 ceph-mon[76349]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:30 smithi184 ceph-mon[100622]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:30 smithi184 ceph-mon[102844]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:30 smithi196 ceph-mon[76349]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:31.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:31.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:31.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:22:32.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:32 smithi184 ceph-mon[100622]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:32.754 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:32 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:32.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:32 smithi184 ceph-mon[102844]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:32.755 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:32 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:32 smithi196 ceph-mon[76349]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:32 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:35 smithi184 ceph-mon[100622]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:35 smithi184 ceph-mon[102844]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:35 smithi196 ceph-mon[76349]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:37.275 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:22:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:37 smithi184 ceph-mon[100622]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:37 smithi184 ceph-mon[102844]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:37 smithi196 ceph-mon[76349]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 5h - - 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 8s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 7s ago 5h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 7s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:22:37.798 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 7s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 8s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 7s ago 5h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 7s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 7s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 7s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 7s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 7s ago 5h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:22:37.799 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:22:37.800 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:22:37.800 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 7s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:22:38.108 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:22:38.109 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:22:38.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:22:38.379 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:22:38.379 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:22:38.379 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:22:38.380 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:22:38.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:38 smithi184 ceph-mon[100622]: from='client.66465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:38.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1126284399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:38.381 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:38 smithi184 ceph-mon[102844]: from='client.66465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:38.381 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1126284399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:38 smithi196 ceph-mon[76349]: from='client.66465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1126284399' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:38.690 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:22:38.691 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:38.691 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:38.691 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:38.691 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[102844]: from='client.76436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[102844]: from='client.76442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[102844]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[102844]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/439686858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[100622]: from='client.76436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[100622]: from='client.76442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[100622]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[100622]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/439686858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:39 smithi196 ceph-mon[76349]: from='client.76436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:39 smithi196 ceph-mon[76349]: from='client.76442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:39 smithi196 ceph-mon[76349]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:39 smithi196 ceph-mon[76349]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/439686858' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:41 smithi184 ceph-mon[100622]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:41 smithi184 ceph-mon[102844]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:41 smithi196 ceph-mon[76349]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:41.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:41.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:41.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:22:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:43 smithi184 ceph-mon[100622]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:43 smithi184 ceph-mon[102844]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:43 smithi196 ceph-mon[76349]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:45.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:45 smithi196 ceph-mon[76349]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:45 smithi184 ceph-mon[100622]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:45 smithi184 ceph-mon[102844]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:47 smithi184 ceph-mon[100622]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:47 smithi184 ceph-mon[102844]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:47.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:47 smithi196 ceph-mon[76349]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[100622]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[102844]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:49 smithi196 ceph-mon[76349]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[100622]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[102844]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:51 smithi196 ceph-mon[76349]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:22:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:51.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:51.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:22:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:22:51.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:22:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:22:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:22:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:53 smithi184 ceph-mon[100622]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:53 smithi184 ceph-mon[102844]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:53.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:53 smithi196 ceph-mon[76349]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:55 smithi184 ceph-mon[100622]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:55 smithi184 ceph-mon[102844]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:55 smithi196 ceph-mon[76349]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:57 smithi184 ceph-mon[100622]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:57 smithi184 ceph-mon[102844]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:57 smithi196 ceph-mon[76349]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:59 smithi184 ceph-mon[100622]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:59 smithi184 ceph-mon[102844]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:59 smithi196 ceph-mon[76349]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:01 smithi184 ceph-mon[100622]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:01 smithi184 ceph-mon[102844]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:01 smithi196 ceph-mon[76349]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:01.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:01.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:01.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:02] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-21T01:23:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:03 smithi184 ceph-mon[100622]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:03 smithi184 ceph-mon[102844]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:03 smithi196 ceph-mon[76349]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:05 smithi184 ceph-mon[100622]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:05 smithi184 ceph-mon[102844]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:05 smithi196 ceph-mon[76349]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:07.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:07 smithi196 ceph-mon[76349]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:07 smithi184 ceph-mon[100622]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:07 smithi184 ceph-mon[102844]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:08.995 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:23:09.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[100622]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:09.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[102844]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:23:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T01:23:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 18s ago 5h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:23:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 5h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:23:09.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:23:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:09 smithi196 ceph-mon[76349]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:09.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:23:09.862 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:23:09.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:23:09.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:09.862 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:23:09.863 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:23:10.153 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:23:10.154 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:23:10.154 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:23:10.154 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:23:10.477 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:23:10.477 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:10.477 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:10.478 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[100622]: from='client.77572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[100622]: from='client.76472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3452672480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[102844]: from='client.77572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[102844]: from='client.76472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3452672480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:10 smithi196 ceph-mon[76349]: from='client.77572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:10 smithi196 ceph-mon[76349]: from='client.76472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3452672480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='client.66513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='client.76490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1281872273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='client.66513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='client.76490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1281872273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='client.66513 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='client.76490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1281872273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:11.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:11.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:11.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:12.274 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:12] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-21T01:23:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:13 smithi184 ceph-mon[100622]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:13 smithi184 ceph-mon[102844]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:13 smithi196 ceph-mon[76349]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:14.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:15 smithi184 ceph-mon[100622]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:15 smithi184 ceph-mon[102844]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:15.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:15 smithi196 ceph-mon[76349]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:17 smithi184 ceph-mon[100622]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:17 smithi184 ceph-mon[102844]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:17 smithi196 ceph-mon[76349]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:19 smithi184 ceph-mon[102844]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:19 smithi184 ceph-mon[100622]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:19 smithi196 ceph-mon[76349]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:21 smithi184 ceph-mon[100622]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:21 smithi184 ceph-mon[102844]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:21 smithi196 ceph-mon[76349]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:21.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:21.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:21.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:23:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:23 smithi184 ceph-mon[100622]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:23 smithi184 ceph-mon[102844]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:23 smithi196 ceph-mon[76349]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:25 smithi196 ceph-mon[76349]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:25 smithi184 ceph-mon[100622]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:25 smithi184 ceph-mon[102844]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:27 smithi196 ceph-mon[76349]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:27 smithi184 ceph-mon[100622]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:27 smithi184 ceph-mon[102844]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:23:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[100622]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[102844]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:29 smithi196 ceph-mon[76349]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:30.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:30.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[100622]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:31.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.517 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[102844]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:31.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:31 smithi196 ceph-mon[76349]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:31.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:31.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:31.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:32.119 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:23:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:33 smithi184 ceph-mon[100622]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:33 smithi184 ceph-mon[102844]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:33 smithi196 ceph-mon[76349]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:35 smithi184 ceph-mon[100622]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:35 smithi184 ceph-mon[102844]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:35 smithi196 ceph-mon[76349]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:37 smithi184 ceph-mon[100622]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:37 smithi184 ceph-mon[102844]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:37 smithi196 ceph-mon[76349]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:39 smithi184 ceph-mon[100622]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:39 smithi184 ceph-mon[102844]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:39 smithi196 ceph-mon[76349]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:40.774 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:23:41.302 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:23:41.302 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 5h - - 2024-09-21T01:23:41.302 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T01:23:41.302 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:23:41.302 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 10s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 10s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:23:41.303 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 10s ago 5h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 10s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 10s ago 5h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 10s ago 5h 66.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:23:41.304 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 10s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:23:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:41 smithi184 ceph-mon[100622]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:41 smithi184 ceph-mon[102844]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:41.612 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:23:41.613 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:23:41.614 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:23:41.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:23:41.614 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:23:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:41 smithi196 ceph-mon[76349]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:23:41.885 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:23:41.886 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:23:41.886 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:23:41.886 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:41.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:41.886 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:41.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:41.886 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:41.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:42.196 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:23:42.196 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:42.196 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:42.197 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:42.197 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:42.197 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:23:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[100622]: from='client.66537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[100622]: from='client.76508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[100622]: from='client.66549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2242951093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4004282996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[102844]: from='client.66537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[102844]: from='client.76508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[102844]: from='client.66549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2242951093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4004282996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:42 smithi196 ceph-mon[76349]: from='client.66537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:42 smithi196 ceph-mon[76349]: from='client.76508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:42 smithi196 ceph-mon[76349]: from='client.66549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2242951093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4004282996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[100622]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[100622]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[102844]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[102844]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:43 smithi196 ceph-mon[76349]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:43 smithi196 ceph-mon[76349]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:45 smithi184 ceph-mon[100622]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:45 smithi184 ceph-mon[102844]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:45 smithi196 ceph-mon[76349]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:47 smithi184 ceph-mon[100622]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:47 smithi184 ceph-mon[102844]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:47 smithi196 ceph-mon[76349]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:49 smithi184 ceph-mon[100622]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:49.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:49 smithi184 ceph-mon[102844]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:49.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:49 smithi196 ceph-mon[76349]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[100622]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[102844]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:51 smithi196 ceph-mon[76349]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:23:51.863 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:51.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:51.911 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:51.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:23:51.911 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:23:51.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:23:52.150 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:23:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:23:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:53 smithi196 ceph-mon[76349]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:53 smithi184 ceph-mon[100622]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:53 smithi184 ceph-mon[102844]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:55 smithi196 ceph-mon[76349]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:55 smithi184 ceph-mon[100622]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:55 smithi184 ceph-mon[102844]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:57 smithi196 ceph-mon[76349]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:57 smithi184 ceph-mon[100622]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:57 smithi184 ceph-mon[102844]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:59 smithi196 ceph-mon[76349]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:59 smithi184 ceph-mon[100622]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:59 smithi184 ceph-mon[102844]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:01 smithi184 ceph-mon[100622]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:01 smithi184 ceph-mon[102844]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:01.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:01.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:01.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:01 smithi196 ceph-mon[76349]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:24:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:03 smithi184 ceph-mon[102844]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:03 smithi184 ceph-mon[100622]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:03 smithi196 ceph-mon[76349]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:05 smithi184 ceph-mon[102844]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:05 smithi184 ceph-mon[100622]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:05 smithi196 ceph-mon[76349]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:07.705 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:07 smithi196 ceph-mon[76349]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:24:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:07 smithi184 ceph-mon[100622]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:24:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:07 smithi184 ceph-mon[102844]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:24:09.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:09 smithi196 ceph-mon[76349]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:24:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:09 smithi184 ceph-mon[100622]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:24:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:09 smithi184 ceph-mon[102844]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[102844]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:10 smithi184 ceph-mon[100622]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:10.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:10.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:10 smithi196 ceph-mon[76349]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:11.928 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:11.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:11.929 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:11.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:11.929 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:11.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:12.277 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.278 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:12.498 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:24:13.060 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:24:13.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 5h 79.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 5h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 5h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 1s ago 5h 68.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:24:13.062 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:24:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:13 smithi184 ceph-mon[100622]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:13 smithi184 ceph-mon[102844]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:13.381 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:24:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:13.383 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:24:13.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:24:13.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:24:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:24:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:13 smithi196 ceph-mon[76349]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:24:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:24:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:24:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:24:13.681 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:24:13.998 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:24:13.999 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:13.999 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='client.76538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3119068396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2471783211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:14.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='client.76538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3119068396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2471783211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:14.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='client.76538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3119068396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2471783211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:15 smithi196 ceph-mon[76349]: from='client.76562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:15 smithi196 ceph-mon[76349]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:24:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:15 smithi184 ceph-mon[100622]: from='client.76562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:15 smithi184 ceph-mon[100622]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:24:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:15 smithi184 ceph-mon[102844]: from='client.76562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:15 smithi184 ceph-mon[102844]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:24:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:17 smithi184 ceph-mon[100622]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:17 smithi184 ceph-mon[102844]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:17 smithi196 ceph-mon[76349]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:19 smithi184 ceph-mon[100622]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T01:24:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:19 smithi184 ceph-mon[102844]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T01:24:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:19 smithi196 ceph-mon[76349]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T01:24:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:21 smithi184 ceph-mon[100622]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:21 smithi184 ceph-mon[102844]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:21 smithi196 ceph-mon[76349]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:21.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:21.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:21.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:24:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:23 smithi184 ceph-mon[100622]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:23 smithi184 ceph-mon[102844]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:23 smithi196 ceph-mon[76349]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:24:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:25 smithi184 ceph-mon[102844]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:24:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:25 smithi184 ceph-mon[100622]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:24:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:25 smithi196 ceph-mon[76349]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T01:24:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:27 smithi184 ceph-mon[100622]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:27 smithi184 ceph-mon[102844]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:27 smithi196 ceph-mon[76349]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:29 smithi184 ceph-mon[100622]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T01:24:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:29 smithi184 ceph-mon[102844]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T01:24:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:29.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:29 smithi196 ceph-mon[76349]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T01:24:29.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:30.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:30.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:30.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:30.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:30 smithi196 ceph-mon[76349]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:24:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[100622]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:30 smithi184 ceph-mon[102844]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:24:31.833 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:31.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 9 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:31.833 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:31.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:31.834 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:31.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:32.233 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:24:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:33 smithi184 ceph-mon[100622]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:24:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:33 smithi184 ceph-mon[102844]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:24:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:33 smithi196 ceph-mon[76349]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:24:34.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:34.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:35 smithi184 ceph-mon[100622]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:24:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:35 smithi184 ceph-mon[102844]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:24:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:35 smithi196 ceph-mon[76349]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:24:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:37 smithi184 ceph-mon[100622]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:24:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:37 smithi184 ceph-mon[102844]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:24:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:37 smithi196 ceph-mon[76349]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:24:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:39 smithi184 ceph-mon[100622]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:39 smithi184 ceph-mon[102844]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:39 smithi196 ceph-mon[76349]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:24:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:41 smithi184 ceph-mon[100622]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:24:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:41 smithi184 ceph-mon[102844]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:24:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:41 smithi196 ceph-mon[76349]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:24:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:41.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:41.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:41.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:24:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:43 smithi184 ceph-mon[100622]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:24:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:43 smithi184 ceph-mon[102844]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:24:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:43 smithi196 ceph-mon[76349]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T01:24:44.295 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:24:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:24:44.828 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 5h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 5h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:24:44.829 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 5h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 5h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 5h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 68.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:24:44.830 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:24:45.135 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:24:45.136 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:24:45.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[100622]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:24:45.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[100622]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:45.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/613850926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:45.409 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[102844]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:24:45.409 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[102844]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:45.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/613850926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:24:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:24:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:24:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:24:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:24:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:24:45.411 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:24:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:45 smithi196 ceph-mon[76349]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:24:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:45 smithi196 ceph-mon[76349]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/613850926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:45.718 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:24:45.719 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:45.719 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:45.719 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:45.719 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[100622]: from='client.66615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[100622]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[100622]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/378521670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[102844]: from='client.66615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[102844]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[102844]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/378521670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:46 smithi196 ceph-mon[76349]: from='client.66615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:46 smithi196 ceph-mon[76349]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:46 smithi196 ceph-mon[76349]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/378521670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:47 smithi184 ceph-mon[100622]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:24:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:47 smithi184 ceph-mon[102844]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:24:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:47 smithi196 ceph-mon[76349]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:24:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:49 smithi184 ceph-mon[100622]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T01:24:49.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:49 smithi184 ceph-mon[102844]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T01:24:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:49 smithi196 ceph-mon[76349]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T01:24:50.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:50.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:50.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:50.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:50 smithi196 ceph-mon[76349]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[100622]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:50 smithi184 ceph-mon[102844]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T01:24:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:51.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:51.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:24:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:24:51.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:24:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:24:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:24:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:52.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:24:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:53 smithi196 ceph-mon[76349]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:53.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:53 smithi184 ceph-mon[100622]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:53.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:53 smithi184 ceph-mon[102844]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:55 smithi184 ceph-mon[100622]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:55 smithi184 ceph-mon[102844]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:55 smithi196 ceph-mon[76349]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:57 smithi196 ceph-mon[76349]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:57 smithi184 ceph-mon[100622]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:57 smithi184 ceph-mon[102844]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:59 smithi196 ceph-mon[76349]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:59 smithi184 ceph-mon[100622]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:59 smithi184 ceph-mon[102844]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:01 smithi196 ceph-mon[76349]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:01 smithi184 ceph-mon[100622]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:01 smithi184 ceph-mon[102844]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:01.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:01.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:01.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:25:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:03 smithi184 ceph-mon[100622]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:03 smithi184 ceph-mon[102844]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:03 smithi196 ceph-mon[76349]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:05 smithi184 ceph-mon[100622]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:05 smithi184 ceph-mon[102844]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:05 smithi196 ceph-mon[76349]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:07 smithi184 ceph-mon[100622]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:07 smithi184 ceph-mon[102844]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:07 smithi196 ceph-mon[76349]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:09.702 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:09 smithi196 ceph-mon[76349]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:09 smithi184 ceph-mon[100622]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:09 smithi184 ceph-mon[102844]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:10 smithi196 ceph-mon[76349]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[100622]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:10 smithi184 ceph-mon[102844]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:11.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:11.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:11.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:25:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[100622]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:12 smithi184 ceph-mon[102844]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:12 smithi196 ceph-mon[76349]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:14.712 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:14.712 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:14 smithi184 ceph-mon[102844]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:14.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:14.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:14 smithi184 ceph-mon[100622]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:14 smithi196 ceph-mon[76349]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:16.006 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:25:16.539 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 6s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 6s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 6s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:25:16.540 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 66.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:25:16.541 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:25:16.849 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:25:16.851 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:25:16.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:25:16.851 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:25:17.116 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:25:17.117 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:25:17.425 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:25:17.425 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:17.425 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:17.425 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[100622]: from='client.76610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[100622]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[100622]: from='client.76616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4175581725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[102844]: from='client.76610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[102844]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[102844]: from='client.76616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4175581725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:17 smithi196 ceph-mon[76349]: from='client.76610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:17 smithi196 ceph-mon[76349]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:17 smithi196 ceph-mon[76349]: from='client.76616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4175581725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[100622]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[100622]: from='client.66669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2470115164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:18.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[102844]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[102844]: from='client.66669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2470115164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:18 smithi196 ceph-mon[76349]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:18 smithi196 ceph-mon[76349]: from='client.66669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2470115164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:19 smithi184 ceph-mon[100622]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:19 smithi184 ceph-mon[102844]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:19 smithi196 ceph-mon[76349]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:21 smithi184 ceph-mon[100622]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:21 smithi184 ceph-mon[102844]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:21 smithi196 ceph-mon[76349]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:21.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:21.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:21.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:25:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:23 smithi184 ceph-mon[100622]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:23 smithi184 ceph-mon[102844]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:23 smithi196 ceph-mon[76349]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:25 smithi184 ceph-mon[100622]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:25 smithi184 ceph-mon[102844]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:25 smithi196 ceph-mon[76349]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:27 smithi184 ceph-mon[100622]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:27 smithi184 ceph-mon[102844]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:27 smithi196 ceph-mon[76349]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:29 smithi184 ceph-mon[100622]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:29 smithi184 ceph-mon[102844]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:29 smithi196 ceph-mon[76349]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:31.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:31 smithi196 ceph-mon[76349]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[100622]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-mon[102844]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:31.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:31.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:31.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:32] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-21T01:25:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[100622]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:32 smithi184 ceph-mon[102844]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:32 smithi196 ceph-mon[76349]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:33.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:33.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:34 smithi196 ceph-mon[76349]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:34.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:34 smithi184 ceph-mon[100622]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:34.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:34 smithi184 ceph-mon[102844]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:37 smithi184 ceph-mon[100622]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:37 smithi184 ceph-mon[102844]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:37 smithi196 ceph-mon[76349]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:39 smithi184 ceph-mon[100622]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:39 smithi184 ceph-mon[102844]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:39 smithi196 ceph-mon[76349]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:41 smithi184 ceph-mon[100622]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:41 smithi184 ceph-mon[102844]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:41 smithi196 ceph-mon[76349]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:41.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:41.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:41.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:42] "GET /metrics HTTP/1.1" 200 38939 "" "Prometheus/2.43.0" 2024-09-21T01:25:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:43 smithi184 ceph-mon[100622]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:43 smithi184 ceph-mon[102844]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:43 smithi196 ceph-mon[76349]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:44.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:45 smithi184 ceph-mon[100622]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:45 smithi184 ceph-mon[102844]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:45 smithi196 ceph-mon[76349]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:47 smithi184 ceph-mon[100622]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:47 smithi184 ceph-mon[102844]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:47 smithi196 ceph-mon[76349]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:47.722 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 18s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 16s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 18s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:25:48.250 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 18s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 18s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 18s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 16s ago 5h 66.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:25:48.251 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:25:48.562 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:25:48.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:25:48.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:25:48.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:48.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:25:48.563 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:25:48.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:25:48.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:25:49.142 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:25:49.143 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:49.143 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:49.143 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:49.143 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: from='client.66681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3175269417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/196315533' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: from='client.66681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3175269417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/196315533' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: from='client.66681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3175269417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/196315533' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[100622]: from='client.66705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[102844]: from='client.66705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:50 smithi196 ceph-mon[76349]: from='client.66705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:51.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:51 smithi196 ceph-mon[76349]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:51.595 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:51 smithi184 ceph-mon[100622]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:51.595 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:51 smithi184 ceph-mon[102844]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:51.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:51.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:25:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:25:51.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:25:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:25:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:25:52.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:52.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:52.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:52.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:52 smithi196 ceph-mon[76349]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[100622]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:52 smithi184 ceph-mon[102844]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:54 smithi184 ceph-mon[100622]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:54 smithi184 ceph-mon[102844]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:54 smithi196 ceph-mon[76349]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:57 smithi184 ceph-mon[100622]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:57 smithi184 ceph-mon[102844]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:57 smithi196 ceph-mon[76349]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:59 smithi184 ceph-mon[100622]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:59 smithi184 ceph-mon[102844]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:59 smithi196 ceph-mon[76349]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:01 smithi184 ceph-mon[100622]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:01.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:01 smithi184 ceph-mon[102844]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:01 smithi196 ceph-mon[76349]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:01.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:01.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:01.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:26:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:03 smithi184 ceph-mon[100622]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:03.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:03 smithi184 ceph-mon[102844]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:03 smithi196 ceph-mon[76349]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:05 smithi196 ceph-mon[76349]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:05 smithi184 ceph-mon[100622]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:05 smithi184 ceph-mon[102844]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:07 smithi196 ceph-mon[76349]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:07 smithi184 ceph-mon[100622]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:07 smithi184 ceph-mon[102844]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:09 smithi196 ceph-mon[76349]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:09 smithi184 ceph-mon[102844]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:09 smithi184 ceph-mon[100622]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:11.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:11 smithi196 ceph-mon[76349]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:11.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[100622]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[102844]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:11.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:11.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:11.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:26:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:12 smithi196 ceph-mon[76349]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[100622]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:13.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:12 smithi184 ceph-mon[102844]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:15.103 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:15.103 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:14 smithi184 ceph-mon[100622]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:15.103 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:15.103 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:14 smithi184 ceph-mon[102844]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:14 smithi196 ceph-mon[76349]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:17 smithi184 ceph-mon[100622]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:17 smithi184 ceph-mon[102844]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:17 smithi196 ceph-mon[76349]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:19.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:19 smithi184 ceph-mon[100622]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:19.430 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:19 smithi184 ceph-mon[102844]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:19.430 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:26:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:19 smithi196 ceph-mon[76349]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:19.954 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 5h - - 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 5h - - 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 9s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 9s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:26:19.955 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:26:19.956 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:26:20.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:26:20.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:26:20.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:26:20.536 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:26:20.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:20 smithi184 ceph-mon[100622]: from='client.66717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:20.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/343447984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:20.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:20 smithi184 ceph-mon[102844]: from='client.66717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:20.537 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/343447984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:20 smithi196 ceph-mon[76349]: from='client.66717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/343447984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:20.847 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:26:20.847 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:20.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:20.848 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:20.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[102844]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[102844]: from='client.76694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[102844]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2411951709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[100622]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[100622]: from='client.76694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[100622]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2411951709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:21 smithi196 ceph-mon[76349]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:21 smithi196 ceph-mon[76349]: from='client.76694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:21 smithi196 ceph-mon[76349]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2411951709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:21.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:21.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:21.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:26:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:22 smithi196 ceph-mon[76349]: from='client.76706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:22 smithi184 ceph-mon[100622]: from='client.76706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:22 smithi184 ceph-mon[102844]: from='client.76706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:23 smithi196 ceph-mon[76349]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:23 smithi184 ceph-mon[100622]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:23 smithi184 ceph-mon[102844]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:25 smithi196 ceph-mon[76349]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:25 smithi184 ceph-mon[100622]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:25 smithi184 ceph-mon[102844]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:27 smithi196 ceph-mon[76349]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:27 smithi184 ceph-mon[100622]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:27 smithi184 ceph-mon[102844]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:29 smithi184 ceph-mon[100622]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:29 smithi184 ceph-mon[102844]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:29 smithi196 ceph-mon[76349]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[100622]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[102844]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:31.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:31.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:31.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:31 smithi196 ceph-mon[76349]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:26:33.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.246 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:32 smithi196 ceph-mon[76349]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[100622]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:32 smithi184 ceph-mon[102844]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:34.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:34.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:34.374 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:35 smithi184 ceph-mon[100622]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:35 smithi184 ceph-mon[102844]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:35 smithi196 ceph-mon[76349]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:37.532 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:37 smithi184 ceph-mon[100622]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:37.532 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:37 smithi184 ceph-mon[102844]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:37 smithi196 ceph-mon[76349]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:39 smithi184 ceph-mon[100622]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:39 smithi184 ceph-mon[102844]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:39 smithi196 ceph-mon[76349]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:41 smithi184 ceph-mon[100622]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:41 smithi184 ceph-mon[102844]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:41 smithi196 ceph-mon[76349]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:41.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:41.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:41.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:26:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:43 smithi196 ceph-mon[76349]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:43 smithi184 ceph-mon[100622]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:43 smithi184 ceph-mon[102844]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:44.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:44.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:45 smithi196 ceph-mon[76349]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:45 smithi184 ceph-mon[100622]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:45 smithi184 ceph-mon[102844]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:47.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:47 smithi196 ceph-mon[76349]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:47 smithi184 ceph-mon[100622]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:47 smithi184 ceph-mon[102844]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:49.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:49 smithi196 ceph-mon[76349]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:49 smithi184 ceph-mon[100622]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:49 smithi184 ceph-mon[102844]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:51.134 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:26:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[100622]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[102844]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:51 smithi196 ceph-mon[76349]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:51.663 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:26:51.663 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 5h - - 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 5h - - 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 1s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 1s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 1s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 1s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 19s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 1s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 1s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 1s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:26:51.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 1s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:26:51.665 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:26:51.974 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:26:51.974 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:26:51.974 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:26:51.975 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:26:51.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:51.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:51.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:51.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:26:51.976 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:26:51.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:26:52.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:26:52.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:26:52.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:26:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='client.66753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/594080085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='client.66753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/594080085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='client.66753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.544 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/594080085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:26:52.593 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:26:52.593 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:52.593 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:52.593 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:52.593 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:53 smithi196 ceph-mon[76349]: from='client.66765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:53 smithi196 ceph-mon[76349]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:53 smithi196 ceph-mon[76349]: from='client.76742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.530 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1073324469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[100622]: from='client.66765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[100622]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[100622]: from='client.76742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1073324469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[102844]: from='client.66765 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[102844]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[102844]: from='client.76742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1073324469' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:54.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:55 smithi196 ceph-mon[76349]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:55 smithi184 ceph-mon[100622]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:55 smithi184 ceph-mon[102844]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:57 smithi184 ceph-mon[102844]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:57 smithi184 ceph-mon[100622]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:57 smithi196 ceph-mon[76349]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:59 smithi184 ceph-mon[100622]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:59 smithi184 ceph-mon[102844]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:59 smithi196 ceph-mon[76349]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:01.600 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:01 smithi184 ceph-mon[100622]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:01.601 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:01 smithi184 ceph-mon[102844]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:01 smithi196 ceph-mon[76349]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:01.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:01.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:01.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:27:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:03 smithi196 ceph-mon[76349]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:03 smithi184 ceph-mon[100622]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:03 smithi184 ceph-mon[102844]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:05 smithi196 ceph-mon[76349]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:05 smithi184 ceph-mon[102844]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:05 smithi184 ceph-mon[100622]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:07 smithi196 ceph-mon[76349]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:07 smithi184 ceph-mon[100622]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:07 smithi184 ceph-mon[102844]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:09 smithi196 ceph-mon[76349]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:09 smithi184 ceph-mon[102844]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:09 smithi184 ceph-mon[100622]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:11 smithi196 ceph-mon[76349]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:11.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[102844]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:11.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:11.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:11.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[100622]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:11.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:27:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:13 smithi196 ceph-mon[76349]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[100622]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[102844]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:14.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:14.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:15.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:15 smithi196 ceph-mon[76349]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:15 smithi184 ceph-mon[100622]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:15 smithi184 ceph-mon[102844]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:17 smithi184 ceph-mon[100622]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:17 smithi184 ceph-mon[102844]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:17 smithi196 ceph-mon[76349]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:19 smithi184 ceph-mon[100622]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:19 smithi184 ceph-mon[102844]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:19 smithi196 ceph-mon[76349]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:21 smithi184 ceph-mon[100622]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:21 smithi184 ceph-mon[102844]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:21.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:21 smithi196 ceph-mon[76349]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:21.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:21.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:21.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:27:22.890 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 5h - - 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 5h - - 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 11s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 11s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:27:23.418 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 11s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 11s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 11s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 11s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 11s ago 5h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:27:23.419 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:27:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:23 smithi184 ceph-mon[100622]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:23 smithi184 ceph-mon[102844]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:23 smithi196 ceph-mon[76349]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:27:23.729 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:27:23.730 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:27:23.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:27:23.999 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:27:24.309 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:27:24.309 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:24.309 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:24.309 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:24.309 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[100622]: from='client.76754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[100622]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[100622]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3016553189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[102844]: from='client.76754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[102844]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[102844]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3016553189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:24 smithi196 ceph-mon[76349]: from='client.76754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:24 smithi196 ceph-mon[76349]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:24 smithi196 ceph-mon[76349]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3016553189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[100622]: from='client.66813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[100622]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3303283953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[102844]: from='client.66813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[102844]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:25.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3303283953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:25 smithi196 ceph-mon[76349]: from='client.66813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:25 smithi196 ceph-mon[76349]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3303283953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:27 smithi184 ceph-mon[100622]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:27.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:27 smithi184 ceph-mon[102844]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:27.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:27 smithi196 ceph-mon[76349]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:29 smithi184 ceph-mon[100622]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:29 smithi184 ceph-mon[102844]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:29 smithi196 ceph-mon[76349]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[100622]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[102844]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.604 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:31 smithi196 ceph-mon[76349]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:31.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:31.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:31.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:27:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:33 smithi196 ceph-mon[76349]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[102844]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[100622]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:34.507 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:35 smithi184 ceph-mon[100622]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:35 smithi184 ceph-mon[102844]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:35 smithi196 ceph-mon[76349]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:37.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:37 smithi184 ceph-mon[100622]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:37.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:37 smithi184 ceph-mon[102844]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:37 smithi196 ceph-mon[76349]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:39 smithi184 ceph-mon[100622]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:39 smithi184 ceph-mon[102844]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:39 smithi196 ceph-mon[76349]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:41 smithi184 ceph-mon[100622]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:41 smithi184 ceph-mon[102844]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:41.604Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:41.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:41.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:41 smithi196 ceph-mon[76349]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:27:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:43 smithi184 ceph-mon[100622]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:43 smithi184 ceph-mon[102844]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:43 smithi196 ceph-mon[76349]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:44.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:45 smithi184 ceph-mon[100622]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:45 smithi184 ceph-mon[102844]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:45 smithi196 ceph-mon[76349]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:47 smithi184 ceph-mon[100622]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:47 smithi184 ceph-mon[102844]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:47 smithi196 ceph-mon[76349]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:27:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:49 smithi184 ceph-mon[100622]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:49 smithi184 ceph-mon[102844]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:49 smithi196 ceph-mon[76349]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[100622]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[102844]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:51.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:51.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:27:51.799 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:27:51.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:27:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:51 smithi196 ceph-mon[76349]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:27:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:27:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[102844]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[100622]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:53 smithi196 ceph-mon[76349]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:27:54.608 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:27:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:55.137 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 5h - - 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 5h - - 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 4s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 4s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 4s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 4s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 4s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 4s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:27:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 4s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 4s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:27:55.139 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:27:55.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:27:55.452 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:27:55.452 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:27:55.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:27:55.452 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:27:55.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:55 smithi196 ceph-mon[76349]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:27:55.719 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:27:55.720 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:27:55.720 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:27:55.720 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:55 smithi184 ceph-mon[102844]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:55.720 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:55 smithi184 ceph-mon[100622]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:56.037 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:27:56.037 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:56.037 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:56.037 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:56.037 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:56.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:56 smithi196 ceph-mon[76349]: from='client.76790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:56 smithi196 ceph-mon[76349]: from='client.76796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:56 smithi196 ceph-mon[76349]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/327944161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:56.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2972567451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[100622]: from='client.76790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[100622]: from='client.76796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[100622]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/327944161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2972567451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[102844]: from='client.76790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[102844]: from='client.76796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[102844]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/327944161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2972567451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:57 smithi184 ceph-mon[100622]: from='client.66849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:57 smithi184 ceph-mon[100622]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:57 smithi184 ceph-mon[102844]: from='client.66849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:57 smithi184 ceph-mon[102844]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:57 smithi196 ceph-mon[76349]: from='client.66849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:57 smithi196 ceph-mon[76349]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.535 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:58 smithi184 ceph-mon[100622]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.535 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:58 smithi184 ceph-mon[102844]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:58 smithi196 ceph-mon[76349]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:59.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:59.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:00.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:00 smithi184 ceph-mon[100622]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:00.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:00 smithi184 ceph-mon[102844]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:00 smithi196 ceph-mon[76349]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:01.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:01.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:01.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:28:03.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:03 smithi196 ceph-mon[76349]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:03 smithi184 ceph-mon[100622]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:03 smithi184 ceph-mon[102844]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:05 smithi196 ceph-mon[76349]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:05 smithi184 ceph-mon[100622]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:05 smithi184 ceph-mon[102844]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:07.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:07 smithi196 ceph-mon[76349]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:07 smithi184 ceph-mon[100622]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:07 smithi184 ceph-mon[102844]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:09 smithi196 ceph-mon[76349]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:09 smithi184 ceph-mon[102844]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:09 smithi184 ceph-mon[100622]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[102844]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[100622]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:11 smithi196 ceph-mon[76349]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:11.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:11.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:11.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:28:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:13 smithi196 ceph-mon[76349]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:28:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[100622]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:28:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[102844]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:28:13.812719863Z level=info msg="Completed cleanup jobs" duration=12.819499ms 2024-09-21T01:28:14.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:14.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:15 smithi196 ceph-mon[76349]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:15 smithi184 ceph-mon[100622]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:15 smithi184 ceph-mon[102844]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:17 smithi196 ceph-mon[76349]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:17 smithi184 ceph-mon[100622]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:17 smithi184 ceph-mon[102844]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:19.544 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:19 smithi184 ceph-mon[100622]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:19.545 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:19 smithi184 ceph-mon[102844]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:19 smithi196 ceph-mon[76349]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:21 smithi196 ceph-mon[76349]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:21.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:21 smithi184 ceph-mon[100622]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:21.796 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:21 smithi184 ceph-mon[102844]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:21.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:21.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:21.796 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:21.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:28:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:23 smithi196 ceph-mon[76349]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:23 smithi184 ceph-mon[100622]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:23 smithi184 ceph-mon[102844]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:25 smithi196 ceph-mon[76349]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:25 smithi184 ceph-mon[100622]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:25 smithi184 ceph-mon[102844]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:26.335 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:28:26.861 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:28:26.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T01:28:26.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 5h - - 2024-09-21T01:28:26.861 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:28:26.861 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 14s ago 5h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 14s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 14s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 14s ago 5h 16.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 14s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:28:26.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 14s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 14s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 14s ago 5h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:28:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:28:27.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:28:27.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:27.172 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:28:27.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:28:27.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:28:27.172 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:28:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:28:27.443 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:28:27.443 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[102844]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:27.443 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[102844]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:27.443 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3845737263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:27.443 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[100622]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:27.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[100622]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:27.444 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3845737263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:27 smithi196 ceph-mon[76349]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:27 smithi196 ceph-mon[76349]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3845737263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:27.753 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:28:27.753 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:27.753 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:28 smithi196 ceph-mon[76349]: from='client.66867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:28 smithi196 ceph-mon[76349]: from='client.66873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:28 smithi196 ceph-mon[76349]: from='client.66885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3039605171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[102844]: from='client.66867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[102844]: from='client.66873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[102844]: from='client.66885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3039605171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[100622]: from='client.66867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[100622]: from='client.66873 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[100622]: from='client.66885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3039605171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:29 smithi196 ceph-mon[76349]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:29 smithi184 ceph-mon[100622]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:29 smithi184 ceph-mon[102844]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[102844]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[100622]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:31 smithi196 ceph-mon[76349]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:31.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:31.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:31.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:28:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:33 smithi196 ceph-mon[76349]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[100622]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[102844]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:35.726 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:35 smithi196 ceph-mon[76349]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:35 smithi184 ceph-mon[100622]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:35 smithi184 ceph-mon[102844]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:37.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:37 smithi196 ceph-mon[76349]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:37 smithi184 ceph-mon[100622]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:37 smithi184 ceph-mon[102844]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:39.688 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:39 smithi184 ceph-mon[100622]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:39.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:39 smithi184 ceph-mon[102844]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:39 smithi196 ceph-mon[76349]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:41 smithi184 ceph-mon[100622]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:41 smithi184 ceph-mon[102844]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:41 smithi196 ceph-mon[76349]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:41.910 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:41.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:41.910 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:41.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:41.910 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:41.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:28:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:43 smithi184 ceph-mon[100622]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:43 smithi184 ceph-mon[102844]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:43 smithi196 ceph-mon[76349]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:45 smithi184 ceph-mon[100622]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:45 smithi184 ceph-mon[102844]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:45.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:45 smithi196 ceph-mon[76349]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:47 smithi184 ceph-mon[100622]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:47 smithi184 ceph-mon[102844]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:47 smithi196 ceph-mon[76349]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:49 smithi184 ceph-mon[100622]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:49 smithi184 ceph-mon[102844]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:49 smithi196 ceph-mon[76349]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:28:51.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[102844]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[100622]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:51 smithi196 ceph-mon[76349]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:51.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:51.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:28:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:28:51.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:28:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:28:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:28:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[100622]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[102844]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:53 smithi196 ceph-mon[76349]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:28:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:55 smithi184 ceph-mon[100622]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:55 smithi184 ceph-mon[102844]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:55 smithi196 ceph-mon[76349]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:57 smithi196 ceph-mon[76349]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:57 smithi184 ceph-mon[100622]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:57 smithi184 ceph-mon[102844]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:58.047 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:28:58.574 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 5h 529M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:28:58.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 7s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:28:58.577 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 7s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:28:58.577 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 7s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:28:58.577 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:28:58.577 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:28:58.577 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:28:58.578 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:28:58.887 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:28:58.887 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:28:58.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:28:58.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:28:58.888 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:28:59.157 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:28:59.157 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:28:59.157 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:28:59.157 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:28:59.157 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:28:59.158 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:28:59.465 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:28:59.466 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:59.466 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:59.466 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:59.466 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[100622]: from='client.66897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[100622]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[100622]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/896407986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[102844]: from='client.66897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[102844]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[102844]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/896407986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:59 smithi196 ceph-mon[76349]: from='client.66897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:59 smithi196 ceph-mon[76349]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:59 smithi196 ceph-mon[76349]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/896407986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[100622]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[100622]: from='client.66921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3459212351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[102844]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[102844]: from='client.66921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3459212351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:00 smithi196 ceph-mon[76349]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:00 smithi196 ceph-mon[76349]: from='client.66921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3459212351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:01 smithi184 ceph-mon[100622]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:01 smithi184 ceph-mon[102844]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:01.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:01.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:01.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:01 smithi196 ceph-mon[76349]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:02.207 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:29:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:03 smithi184 ceph-mon[100622]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:03 smithi184 ceph-mon[102844]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:03 smithi196 ceph-mon[76349]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:05 smithi184 ceph-mon[100622]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:05 smithi184 ceph-mon[102844]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:05 smithi196 ceph-mon[76349]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:06 smithi184 ceph-mon[100622]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:06 smithi184 ceph-mon[102844]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:06 smithi196 ceph-mon[76349]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:09 smithi196 ceph-mon[76349]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:09 smithi184 ceph-mon[100622]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:09 smithi184 ceph-mon[102844]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:11.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:11 smithi184 ceph-mon[100622]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:11.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:11 smithi184 ceph-mon[102844]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:11 smithi196 ceph-mon[76349]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:11.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:11.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:11.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:29:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[100622]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[102844]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:13 smithi196 ceph-mon[76349]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:14.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:15 smithi184 ceph-mon[100622]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:15 smithi184 ceph-mon[102844]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:15 smithi196 ceph-mon[76349]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:17.576 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:17 smithi196 ceph-mon[76349]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:17 smithi184 ceph-mon[100622]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:17 smithi184 ceph-mon[102844]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:19 smithi184 ceph-mon[100622]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:19 smithi184 ceph-mon[102844]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:19 smithi196 ceph-mon[76349]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:21 smithi184 ceph-mon[102844]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:21 smithi184 ceph-mon[100622]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:21.865 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:21.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:21.865 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:21.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:21.865 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:21.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:21 smithi196 ceph-mon[76349]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:22.135 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:22] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-21T01:29:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:23 smithi184 ceph-mon[100622]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:23 smithi184 ceph-mon[102844]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:23 smithi196 ceph-mon[76349]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:25 smithi184 ceph-mon[100622]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:25 smithi184 ceph-mon[102844]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:25 smithi196 ceph-mon[76349]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:26 smithi184 ceph-mon[100622]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:26 smithi184 ceph-mon[102844]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:26 smithi196 ceph-mon[76349]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:29 smithi196 ceph-mon[76349]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:29.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:29 smithi184 ceph-mon[100622]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:29.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:29.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:29 smithi184 ceph-mon[102844]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:29.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:29.760 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 18s ago 5h 529M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:29:30.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 18s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 18s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 18s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 18s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 17s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 17s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:29:30.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:29:30.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:29:30.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:29:30.602 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:29:30.602 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:29:30.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:29:30.602 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:29:30.870 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:29:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:29:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:29:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:29:30.871 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:29:31.179 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:29:31.179 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:31.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:31.179 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:31.179 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: from='client.76898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: from='client.76904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: from='client.66945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2034620964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/469873801' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: from='client.76898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: from='client.76904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: from='client.66945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2034620964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/469873801' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: from='client.76898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: from='client.76904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: from='client.66945 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2034620964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/469873801' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:31.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:31.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:31.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:32] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:29:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:32 smithi196 ceph-mon[76349]: from='client.66957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[100622]: from='client.66957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[102844]: from='client.66957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:32.674 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[100622]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[102844]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:33 smithi196 ceph-mon[76349]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:35 smithi196 ceph-mon[76349]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:35 smithi184 ceph-mon[100622]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:35 smithi184 ceph-mon[102844]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:37.474 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:37 smithi196 ceph-mon[76349]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:37 smithi184 ceph-mon[100622]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:37 smithi184 ceph-mon[102844]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:39 smithi196 ceph-mon[76349]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:39 smithi184 ceph-mon[100622]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:39 smithi184 ceph-mon[102844]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:41.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:41 smithi184 ceph-mon[100622]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:41.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:41 smithi184 ceph-mon[102844]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:41 smithi196 ceph-mon[76349]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:41.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:41.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:41.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:41.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:41.987 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:41.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:42] "GET /metrics HTTP/1.1" 200 38951 "" "Prometheus/2.43.0" 2024-09-21T01:29:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:43 smithi196 ceph-mon[76349]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:43 smithi184 ceph-mon[100622]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:43 smithi184 ceph-mon[102844]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:45 smithi184 ceph-mon[102844]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:45 smithi184 ceph-mon[100622]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:45 smithi196 ceph-mon[76349]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:47 smithi184 ceph-mon[102844]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:47 smithi184 ceph-mon[100622]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:47 smithi196 ceph-mon[76349]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:49 smithi184 ceph-mon[102844]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:49 smithi184 ceph-mon[100622]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:49.834 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:49 smithi196 ceph-mon[76349]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:51.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:51 smithi184 ceph-mon[102844]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:51.682 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:51 smithi184 ceph-mon[100622]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:51.682 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:51.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:51.682 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:51.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:29:51.682 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:29:51.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:29:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:51 smithi196 ceph-mon[76349]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:29:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:29:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[100622]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:52 smithi184 ceph-mon[102844]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:52 smithi196 ceph-mon[76349]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:29:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:55 smithi196 ceph-mon[76349]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:55 smithi184 ceph-mon[100622]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:55 smithi184 ceph-mon[102844]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:57.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:57 smithi184 ceph-mon[100622]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:57.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:57 smithi184 ceph-mon[102844]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:57 smithi196 ceph-mon[76349]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:59.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:59 smithi196 ceph-mon[76349]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:59.475 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:59 smithi184 ceph-mon[100622]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:59 smithi184 ceph-mon[102844]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:01.481 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:30:01.615 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:01 smithi184 ceph-mon[100622]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:01.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:01 smithi184 ceph-mon[102844]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:01 smithi196 ceph-mon[76349]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:01.980 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:01.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:01.980 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:01.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:01.980 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:01.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:02.016 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:30:02.016 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 5h - - 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 5h - - 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 10s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 10s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 10s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 10s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 10s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 10s ago 5h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 10s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 10s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 10s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 10s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:30:02.017 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:30:02.018 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:30:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:30:02.338 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:30:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:30:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:30:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:30:02.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:30:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:30:02.613 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:30:02.923 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:02.924 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:02.924 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:02.924 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:02.924 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: from='client.76934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: from='client.66975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: from='client.76946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/680551128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:03.489 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1565902859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: from='client.76934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: from='client.66975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: from='client.76946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/680551128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:03.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1565902859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: from='client.76934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: from='client.66975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: from='client.76946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/680551128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1565902859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:04 smithi184 ceph-mon[102844]: from='client.76958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:04 smithi184 ceph-mon[100622]: from='client.76958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:04 smithi196 ceph-mon[76349]: from='client.76958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:05 smithi196 ceph-mon[76349]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:05 smithi184 ceph-mon[100622]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:05 smithi184 ceph-mon[102844]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:07 smithi196 ceph-mon[76349]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:07 smithi184 ceph-mon[100622]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:07 smithi184 ceph-mon[102844]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:09 smithi196 ceph-mon[76349]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:09 smithi184 ceph-mon[100622]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:09 smithi184 ceph-mon[102844]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:11.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:11 smithi184 ceph-mon[100622]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:11.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:11 smithi184 ceph-mon[102844]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:11 smithi196 ceph-mon[76349]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:11.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:11.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:11.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:30:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[102844]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:12 smithi184 ceph-mon[100622]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:12 smithi196 ceph-mon[76349]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:15 smithi196 ceph-mon[76349]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:15 smithi184 ceph-mon[100622]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:15 smithi184 ceph-mon[102844]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:17 smithi184 ceph-mon[100622]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:17 smithi184 ceph-mon[102844]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:17 smithi196 ceph-mon[76349]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:19.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:19 smithi196 ceph-mon[76349]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:19 smithi184 ceph-mon[100622]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:19 smithi184 ceph-mon[102844]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:21 smithi184 ceph-mon[100622]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:21 smithi184 ceph-mon[102844]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:21.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:21.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:21.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:21 smithi196 ceph-mon[76349]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:30:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:22 smithi184 ceph-mon[100622]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:22 smithi184 ceph-mon[102844]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:22 smithi196 ceph-mon[76349]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:24.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:24 smithi184 ceph-mon[102844]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:24.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:24 smithi184 ceph-mon[100622]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:24 smithi196 ceph-mon[76349]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:27 smithi196 ceph-mon[76349]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:27 smithi184 ceph-mon[100622]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:27 smithi184 ceph-mon[102844]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:29 smithi196 ceph-mon[76349]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:29 smithi184 ceph-mon[100622]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:29 smithi184 ceph-mon[102844]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:31 smithi196 ceph-mon[76349]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:31 smithi184 ceph-mon[100622]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:31 smithi184 ceph-mon[102844]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:31.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:31.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:31.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:30:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[102844]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:32 smithi184 ceph-mon[100622]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:33.220 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:30:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:32 smithi196 ceph-mon[76349]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:33.746 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 5h - - 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 5h - - 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 1s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 0s ago 5h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 1s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 0s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 1s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 1s ago 5h 529M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 0s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 1s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 1s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:30:33.747 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 0s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 1s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 1s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 0s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 0s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 0s ago 5h 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:30:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 0s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[100622]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[102844]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:30:34.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:30:34.058 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:30:34.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:30:34.334 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:30:34.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:33 smithi196 ceph-mon[76349]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:34.646 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:34.647 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:34.647 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:34.647 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:34.647 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: from='client.76976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3104524500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: from='client.76994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1969886222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: from='client.76976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3104524500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: from='client.76994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1969886222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: from='client.76976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3104524500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: from='client.76994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1969886222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:37 smithi196 ceph-mon[76349]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:37 smithi184 ceph-mon[100622]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:37 smithi184 ceph-mon[102844]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:39.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:39 smithi196 ceph-mon[76349]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:39 smithi184 ceph-mon[100622]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:39 smithi184 ceph-mon[102844]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:41 smithi196 ceph-mon[76349]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:41 smithi184 ceph-mon[100622]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:41 smithi184 ceph-mon[102844]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:41.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:41.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:41.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:30:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:43 smithi184 ceph-mon[102844]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:43 smithi184 ceph-mon[100622]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:43 smithi196 ceph-mon[76349]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:44.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:44.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:45.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:45 smithi196 ceph-mon[76349]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:45.723 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:45 smithi184 ceph-mon[100622]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:45.723 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:45 smithi184 ceph-mon[102844]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:47 smithi184 ceph-mon[102844]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:47 smithi184 ceph-mon[100622]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:47 smithi196 ceph-mon[76349]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:49 smithi184 ceph-mon[100622]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:49 smithi184 ceph-mon[102844]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:49 smithi196 ceph-mon[76349]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:51 smithi184 ceph-mon[100622]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:51 smithi184 ceph-mon[102844]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:51.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:51.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:30:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:30:51.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:51 smithi196 ceph-mon[76349]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:30:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:30:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:53 smithi196 ceph-mon[76349]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[100622]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:53.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:53 smithi184 ceph-mon[102844]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:30:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:55 smithi196 ceph-mon[76349]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:55 smithi184 ceph-mon[100622]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:55 smithi184 ceph-mon[102844]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:57 smithi196 ceph-mon[76349]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:57 smithi184 ceph-mon[100622]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:57 smithi184 ceph-mon[102844]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:59 smithi196 ceph-mon[76349]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:59 smithi184 ceph-mon[100622]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:59 smithi184 ceph-mon[102844]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:01 smithi196 ceph-mon[76349]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:01 smithi184 ceph-mon[100622]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:01 smithi184 ceph-mon[102844]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:01.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:01.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:01.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:31:03.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:03 smithi184 ceph-mon[100622]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:03.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:03 smithi184 ceph-mon[102844]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:03.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:03 smithi196 ceph-mon[76349]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:04.933 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 5h - - 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 5h - - 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 13s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 11s ago 5h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 13s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 11s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 13s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 13s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:31:05.491 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 11s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 13s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 13s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 11s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 13s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 13s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 13s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 13s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 11s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 11s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 11s ago 5h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:31:05.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:31:05.493 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:31:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:05 smithi184 ceph-mon[100622]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:05 smithi184 ceph-mon[102844]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:05.816 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:31:05.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:31:05.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:05.818 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:31:05.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:31:05.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:31:05.818 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:31:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:05 smithi196 ceph-mon[76349]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:06.091 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:31:06.092 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:31:06.093 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:31:06.094 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:31:06.414 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:31:06.414 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:06.414 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:06.414 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:06.414 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:06.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[100622]: from='client.77006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[100622]: from='client.77012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[100622]: from='client.67053 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.792 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2180944264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:06.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[102844]: from='client.77006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[102844]: from='client.77012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[102844]: from='client.67053 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2180944264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:06 smithi196 ceph-mon[76349]: from='client.77006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:06 smithi196 ceph-mon[76349]: from='client.77012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:06 smithi196 ceph-mon[76349]: from='client.67053 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2180944264' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[100622]: from='client.67065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[100622]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/55325169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[102844]: from='client.67065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[102844]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/55325169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:07 smithi196 ceph-mon[76349]: from='client.67065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:07 smithi196 ceph-mon[76349]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/55325169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:09 smithi184 ceph-mon[100622]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:09 smithi184 ceph-mon[102844]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:09 smithi196 ceph-mon[76349]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:11 smithi184 ceph-mon[100622]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:11 smithi184 ceph-mon[102844]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:11.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:11.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:11.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:11 smithi196 ceph-mon[76349]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:31:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[100622]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:13 smithi184 ceph-mon[102844]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:13.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:13 smithi196 ceph-mon[76349]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:15 smithi184 ceph-mon[102844]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:15 smithi184 ceph-mon[100622]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:15 smithi196 ceph-mon[76349]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:17 smithi184 ceph-mon[100622]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:17 smithi184 ceph-mon[102844]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:17 smithi196 ceph-mon[76349]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:19 smithi184 ceph-mon[100622]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:19 smithi184 ceph-mon[102844]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:19 smithi196 ceph-mon[76349]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:21.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:21 smithi196 ceph-mon[76349]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:21 smithi184 ceph-mon[100622]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:21 smithi184 ceph-mon[102844]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:21.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:21.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:21.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:31:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:23 smithi184 ceph-mon[100622]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:23 smithi184 ceph-mon[102844]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:23 smithi196 ceph-mon[76349]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:24.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:24.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:25 smithi184 ceph-mon[100622]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:25 smithi184 ceph-mon[102844]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:25 smithi196 ceph-mon[76349]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:27 smithi184 ceph-mon[100622]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:27 smithi184 ceph-mon[102844]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:27 smithi196 ceph-mon[76349]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:29 smithi184 ceph-mon[100622]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:29 smithi184 ceph-mon[102844]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:29 smithi196 ceph-mon[76349]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:31 smithi184 ceph-mon[100622]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:31 smithi184 ceph-mon[102844]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:31.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:31.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:31.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:31 smithi196 ceph-mon[76349]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:31:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:33 smithi196 ceph-mon[76349]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[102844]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:33 smithi184 ceph-mon[100622]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:35 smithi196 ceph-mon[76349]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:35 smithi184 ceph-mon[100622]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:35 smithi184 ceph-mon[102844]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:36.705 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 5h - - 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 4s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 3s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 4s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:31:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 4s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 4s ago 5h 529M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 4s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 4s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 3s ago 5h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 4s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 4s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 3s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 3s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 5h 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:31:37.229 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:31:37.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:31:37.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:31:37.230 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:31:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:31:37.542 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:31:37.543 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:37 smithi184 ceph-mon[100622]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:37.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:37 smithi184 ceph-mon[102844]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:31:37.810 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:31:37.811 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:31:37.811 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:31:37.811 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:31:37.811 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:31:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:37 smithi196 ceph-mon[76349]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:38.123 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:31:38.123 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:38.124 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:38.124 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:38.124 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[102844]: from='client.77042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[102844]: from='client.67083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[102844]: from='client.67089 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2345331708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/145551116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[100622]: from='client.77042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[100622]: from='client.67083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[100622]: from='client.67089 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2345331708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/145551116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:38 smithi196 ceph-mon[76349]: from='client.77042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:38 smithi196 ceph-mon[76349]: from='client.67083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:38 smithi196 ceph-mon[76349]: from='client.67089 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2345331708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/145551116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:39 smithi184 ceph-mon[100622]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:39 smithi184 ceph-mon[100622]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:39 smithi184 ceph-mon[102844]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:39 smithi184 ceph-mon[102844]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:39 smithi196 ceph-mon[76349]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:39 smithi196 ceph-mon[76349]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:41 smithi184 ceph-mon[100622]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:41 smithi184 ceph-mon[102844]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:41.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:41.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:41.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:41.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:41 smithi196 ceph-mon[76349]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:31:43.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:43 smithi196 ceph-mon[76349]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:43 smithi184 ceph-mon[100622]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:43 smithi184 ceph-mon[102844]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:45.606 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:45 smithi184 ceph-mon[100622]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:45.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:45 smithi184 ceph-mon[102844]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:45 smithi196 ceph-mon[76349]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:47 smithi184 ceph-mon[100622]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:47 smithi184 ceph-mon[102844]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:47 smithi196 ceph-mon[76349]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:49 smithi184 ceph-mon[100622]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:49 smithi184 ceph-mon[102844]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:49 smithi196 ceph-mon[76349]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:51 smithi184 ceph-mon[100622]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:51 smithi184 ceph-mon[102844]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:51.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:31:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:31:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:31:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:51 smithi196 ceph-mon[76349]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:31:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:31:53.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:53 smithi196 ceph-mon[76349]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:53.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[100622]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[102844]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:31:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:55 smithi184 ceph-mon[100622]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:55 smithi184 ceph-mon[102844]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:55 smithi196 ceph-mon[76349]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:57.736 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:57 smithi184 ceph-mon[100622]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:57.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:57 smithi184 ceph-mon[102844]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:57 smithi196 ceph-mon[76349]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:59 smithi184 ceph-mon[100622]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:59 smithi184 ceph-mon[102844]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:59 smithi196 ceph-mon[76349]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:01.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:01 smithi196 ceph-mon[76349]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:01 smithi184 ceph-mon[100622]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:01 smithi184 ceph-mon[102844]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:01.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:01.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:01.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:32:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:03 smithi184 ceph-mon[100622]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:03 smithi184 ceph-mon[102844]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:03 smithi196 ceph-mon[76349]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:05 smithi184 ceph-mon[100622]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:05 smithi184 ceph-mon[102844]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:05 smithi196 ceph-mon[76349]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:07 smithi184 ceph-mon[102844]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:07 smithi184 ceph-mon[100622]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:07 smithi196 ceph-mon[76349]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:08.419 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 5h - - 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 5h - - 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 16s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 14s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:32:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 16s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 14s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 16s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 16s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 14s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 16s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 16s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 14s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 16s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 16s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 16s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:32:08.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 16s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 14s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 14s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 14s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 14s ago 5h 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:32:08.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:32:09.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:32:09.270 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:32:09.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[102844]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:09.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[102844]: from='client.67113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:09.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1193004549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:09.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[100622]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:09.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[100622]: from='client.67113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:09.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1193004549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:32:09.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:32:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:32:09.850 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:32:09.851 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:09.851 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:09.851 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:09.851 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:09 smithi196 ceph-mon[76349]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:09 smithi196 ceph-mon[76349]: from='client.67113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1193004549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[100622]: from='client.77084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[100622]: from='client.77090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2773897083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[102844]: from='client.77084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[102844]: from='client.77090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2773897083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:10 smithi196 ceph-mon[76349]: from='client.77084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:10 smithi196 ceph-mon[76349]: from='client.77090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2773897083' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-mon[100622]: from='client.67137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-mon[100622]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-mon[102844]: from='client.67137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-mon[102844]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:11.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:11.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:11.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:11 smithi196 ceph-mon[76349]: from='client.67137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:11 smithi196 ceph-mon[76349]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:32:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:12 smithi184 ceph-mon[100622]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:12 smithi184 ceph-mon[102844]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:12 smithi196 ceph-mon[76349]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:14 smithi196 ceph-mon[76349]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[100622]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:14 smithi184 ceph-mon[102844]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:17 smithi184 ceph-mon[100622]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:17 smithi184 ceph-mon[102844]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:17 smithi196 ceph-mon[76349]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:19 smithi184 ceph-mon[100622]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:19 smithi184 ceph-mon[102844]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:19 smithi196 ceph-mon[76349]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:21 smithi184 ceph-mon[100622]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:21 smithi184 ceph-mon[102844]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:21.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:21.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:21.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:21 smithi196 ceph-mon[76349]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:32:23.707 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:23 smithi196 ceph-mon[76349]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:23 smithi184 ceph-mon[100622]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:23 smithi184 ceph-mon[102844]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:24.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:25 smithi196 ceph-mon[76349]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:25 smithi184 ceph-mon[100622]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:25 smithi184 ceph-mon[102844]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:27.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:27 smithi184 ceph-mon[100622]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:27.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:27 smithi184 ceph-mon[102844]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:27 smithi196 ceph-mon[76349]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:29.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:29 smithi184 ceph-mon[102844]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:29.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:29.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:29 smithi184 ceph-mon[100622]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:29.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:29 smithi196 ceph-mon[76349]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:31 smithi184 ceph-mon[100622]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:31 smithi184 ceph-mon[102844]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:31.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:31.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:31.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:31 smithi196 ceph-mon[76349]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:32:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[100622]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[102844]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:33 smithi196 ceph-mon[76349]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:35 smithi184 ceph-mon[100622]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:35 smithi184 ceph-mon[102844]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:35 smithi196 ceph-mon[76349]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:37 smithi184 ceph-mon[100622]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:37 smithi184 ceph-mon[102844]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:37.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:37 smithi196 ceph-mon[76349]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:39 smithi184 ceph-mon[100622]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:39 smithi184 ceph-mon[102844]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:39 smithi196 ceph-mon[76349]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:40.135 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:32:40.662 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:32:40.662 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 5h - - 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 6s ago 5h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 6s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 7s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 7s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:32:40.663 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 7s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 6s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 6s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 6s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 5h 72.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:32:40.664 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:32:40.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:32:40.970 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:32:41.235 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:32:41.236 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[102844]: from='client.77114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[102844]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[102844]: from='client.67155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1714926085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[100622]: from='client.77114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[100622]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[100622]: from='client.67155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1714926085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:41.549 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:32:41.549 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:41.549 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:41.549 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:41.549 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:41 smithi196 ceph-mon[76349]: from='client.77114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:41 smithi196 ceph-mon[76349]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:41 smithi196 ceph-mon[76349]: from='client.67155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1714926085' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:41.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:41.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:41.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:32:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[100622]: from='client.77126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[100622]: from='client.77138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3212452527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[102844]: from='client.77126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[102844]: from='client.77138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3212452527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:42 smithi196 ceph-mon[76349]: from='client.77126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:42 smithi196 ceph-mon[76349]: from='client.77138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3212452527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:43.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:43 smithi196 ceph-mon[76349]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:43 smithi184 ceph-mon[100622]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:43 smithi184 ceph-mon[102844]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:45.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:45 smithi196 ceph-mon[76349]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:45 smithi184 ceph-mon[100622]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:45 smithi184 ceph-mon[102844]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:47.771 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:47 smithi184 ceph-mon[100622]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:47.772 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:47 smithi184 ceph-mon[102844]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:47 smithi196 ceph-mon[76349]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:49 smithi184 ceph-mon[100622]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:49 smithi184 ceph-mon[102844]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:49 smithi196 ceph-mon[76349]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:51 smithi184 ceph-mon[100622]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:51 smithi184 ceph-mon[102844]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:51.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:51.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:32:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:32:51.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:32:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:51 smithi196 ceph-mon[76349]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:32:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:32:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[100622]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[102844]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:53 smithi196 ceph-mon[76349]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:32:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:55 smithi184 ceph-mon[100622]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:55 smithi184 ceph-mon[102844]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:55 smithi196 ceph-mon[76349]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:56 smithi184 ceph-mon[100622]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:56 smithi184 ceph-mon[102844]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:56 smithi196 ceph-mon[76349]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:32:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:59 smithi184 ceph-mon[100622]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:59 smithi184 ceph-mon[102844]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:59 smithi196 ceph-mon[76349]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:01 smithi184 ceph-mon[102844]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:01 smithi184 ceph-mon[100622]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:01.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:01.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:01.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:01 smithi196 ceph-mon[76349]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:33:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:03 smithi184 ceph-mon[100622]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:03 smithi184 ceph-mon[102844]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:03 smithi196 ceph-mon[76349]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:04.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:05 smithi196 ceph-mon[76349]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:05 smithi184 ceph-mon[100622]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:05 smithi184 ceph-mon[102844]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:07 smithi196 ceph-mon[76349]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:07 smithi184 ceph-mon[100622]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:07 smithi184 ceph-mon[102844]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:09.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:09 smithi184 ceph-mon[102844]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:09.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:09 smithi184 ceph-mon[100622]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:09 smithi196 ceph-mon[76349]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:11 smithi184 ceph-mon[100622]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:11 smithi184 ceph-mon[102844]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:11.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:11.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:11.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:11.849 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:33:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:11 smithi196 ceph-mon[76349]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:33:12.375 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 5h - - 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 5h - - 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 19s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 93.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 19s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:33:12.376 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 17s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 17s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:33:12.377 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:33:12.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:33:12.684 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:33:12.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:33:12.686 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:33:12.956 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:33:12.956 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:33:12.956 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:33:12.957 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:33:13.268 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:33:13.268 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:13.268 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:13.268 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:13.268 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='client.77150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='client.77162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:13.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1425869453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3106067924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='client.77150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='client.77162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1425869453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3106067924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='client.77150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='client.77162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1425869453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3106067924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:14 smithi196 ceph-mon[76349]: from='client.67209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[100622]: from='client.67209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[102844]: from='client.67209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[100622]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[102844]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:15 smithi196 ceph-mon[76349]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:17 smithi184 ceph-mon[100622]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:17 smithi184 ceph-mon[102844]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:17 smithi196 ceph-mon[76349]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:19 smithi184 ceph-mon[100622]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:19 smithi184 ceph-mon[102844]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:19 smithi196 ceph-mon[76349]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:21 smithi184 ceph-mon[100622]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:21 smithi184 ceph-mon[102844]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:21.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:21.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:21.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:21 smithi196 ceph-mon[76349]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:33:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:23 smithi184 ceph-mon[100622]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:23 smithi184 ceph-mon[102844]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:23 smithi196 ceph-mon[76349]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:24.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:25 smithi196 ceph-mon[76349]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:25 smithi184 ceph-mon[100622]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:25 smithi184 ceph-mon[102844]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:27 smithi184 ceph-mon[100622]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:27 smithi184 ceph-mon[102844]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:27 smithi196 ceph-mon[76349]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:29.712 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:29 smithi184 ceph-mon[100622]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:29.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:29.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:29 smithi184 ceph-mon[102844]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:29.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:29 smithi196 ceph-mon[76349]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:31.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:31 smithi184 ceph-mon[100622]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:31.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:31 smithi184 ceph-mon[102844]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:31 smithi196 ceph-mon[76349]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:31.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:31.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:31.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:33:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[100622]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[102844]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:33 smithi196 ceph-mon[76349]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[100622]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[102844]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:35 smithi196 ceph-mon[76349]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:36 smithi184 ceph-mon[100622]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:36 smithi184 ceph-mon[102844]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:36 smithi196 ceph-mon[76349]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:39 smithi184 ceph-mon[100622]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:39 smithi184 ceph-mon[102844]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:39 smithi196 ceph-mon[76349]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:41 smithi184 ceph-mon[100622]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:41 smithi184 ceph-mon[102844]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:41.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:41.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:41.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:41 smithi196 ceph-mon[76349]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:33:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:43 smithi184 ceph-mon[100622]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:43 smithi184 ceph-mon[102844]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:43.559 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:33:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:43 smithi196 ceph-mon[76349]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 5h - - 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 11s ago 5h 28.0M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 9s ago 5h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 11s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 11s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 11s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 11s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 9s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:33:44.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 11s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 11s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 11s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 11s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 9s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 9s ago 5h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 5h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:33:44.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 9s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:44.404 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:33:44.405 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:33:44.677 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:33:44.678 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1292536768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:44.679 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1292536768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1292536768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:33:44.988 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:44.989 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:44.989 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:44.989 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:45.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:45 smithi196 ceph-mon[76349]: from='client.67221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:45 smithi196 ceph-mon[76349]: from='client.77192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:45 smithi196 ceph-mon[76349]: from='client.77198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:45 smithi196 ceph-mon[76349]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:45.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1947234992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[100622]: from='client.67221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[100622]: from='client.77192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[100622]: from='client.77198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[100622]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1947234992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[102844]: from='client.67221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[102844]: from='client.77192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[102844]: from='client.77198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[102844]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1947234992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:46.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:46 smithi196 ceph-mon[76349]: from='client.67245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:46 smithi184 ceph-mon[102844]: from='client.67245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:46 smithi184 ceph-mon[100622]: from='client.67245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:47 smithi184 ceph-mon[102844]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:47 smithi184 ceph-mon[100622]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:47.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:47 smithi196 ceph-mon[76349]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:49.698 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:49 smithi184 ceph-mon[100622]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:49.698 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:49 smithi184 ceph-mon[102844]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:49 smithi196 ceph-mon[76349]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:51.521 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:51 smithi184 ceph-mon[100622]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:51.522 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:51 smithi184 ceph-mon[102844]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:51.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:51.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:33:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:33:51.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:33:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:51 smithi196 ceph-mon[76349]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:52.144 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:33:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:33:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[100622]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[102844]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:53 smithi196 ceph-mon[76349]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:54.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[100622]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[102844]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:55 smithi196 ceph-mon[76349]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:33:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:57 smithi184 ceph-mon[100622]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:57 smithi184 ceph-mon[102844]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:57 smithi196 ceph-mon[76349]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:59 smithi184 ceph-mon[100622]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:59 smithi184 ceph-mon[102844]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:59 smithi196 ceph-mon[76349]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:01 smithi184 ceph-mon[100622]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:01 smithi184 ceph-mon[102844]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:01.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:01.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:01.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:01 smithi196 ceph-mon[76349]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:34:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:03 smithi184 ceph-mon[100622]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:03 smithi184 ceph-mon[102844]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:03 smithi196 ceph-mon[76349]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:05 smithi184 ceph-mon[100622]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:05 smithi184 ceph-mon[102844]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:05 smithi196 ceph-mon[76349]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:06 smithi196 ceph-mon[76349]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:06 smithi184 ceph-mon[100622]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:06 smithi184 ceph-mon[102844]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:09 smithi184 ceph-mon[100622]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:09 smithi184 ceph-mon[102844]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:09 smithi196 ceph-mon[76349]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:11 smithi184 ceph-mon[100622]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:11.624 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:11 smithi184 ceph-mon[102844]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:11 smithi196 ceph-mon[76349]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:11.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:11.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:11.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[100622]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[102844]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:13 smithi196 ceph-mon[76349]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:34:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:15.275 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[100622]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[102844]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:34:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:15 smithi196 ceph-mon[76349]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T01:34:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 5h - - 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 5h - - 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 0s ago 5h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:34:15.811 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 0s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 0s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 0s ago 5h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 5h 92.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 5h 96.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 0s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 0s ago 5h 99.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:34:15.812 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 0s ago 5h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:34:15.813 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:34:15.813 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:34:15.813 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:34:15.813 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 0s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:16.122 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:34:16.123 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:34:16.391 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:34:16.391 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:34:16.391 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:34:16.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:34:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:34:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:16.706 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:16.706 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:16.706 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:16 smithi184 ceph-mon[100622]: from='client.67257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:16.706 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/262944272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:16.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:16 smithi184 ceph-mon[102844]: from='client.67257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:16.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/262944272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:16 smithi196 ceph-mon[76349]: from='client.67257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/262944272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[100622]: from='client.77228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[100622]: from='client.77234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[100622]: from='client.67281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[100622]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:34:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3945138933' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[102844]: from='client.77228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[102844]: from='client.77234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[102844]: from='client.67281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[102844]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:34:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3945138933' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:17 smithi196 ceph-mon[76349]: from='client.77228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:17 smithi196 ceph-mon[76349]: from='client.77234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:17 smithi196 ceph-mon[76349]: from='client.67281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:17 smithi196 ceph-mon[76349]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:34:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3945138933' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:19 smithi184 ceph-mon[100622]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T01:34:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:19 smithi184 ceph-mon[102844]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T01:34:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:19 smithi196 ceph-mon[76349]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T01:34:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:21 smithi184 ceph-mon[100622]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T01:34:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:21 smithi184 ceph-mon[102844]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T01:34:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:21.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:21.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:21.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:21 smithi196 ceph-mon[76349]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T01:34:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:34:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:23 smithi184 ceph-mon[102844]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:34:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:23 smithi184 ceph-mon[100622]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:34:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:23 smithi196 ceph-mon[76349]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:34:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:25 smithi184 ceph-mon[100622]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:34:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:25 smithi184 ceph-mon[102844]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:34:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:25 smithi196 ceph-mon[76349]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:34:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:27 smithi184 ceph-mon[100622]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T01:34:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:27 smithi184 ceph-mon[102844]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T01:34:27.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:27 smithi196 ceph-mon[76349]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T01:34:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:29 smithi184 ceph-mon[100622]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T01:34:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:29 smithi184 ceph-mon[102844]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T01:34:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:29.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:29 smithi196 ceph-mon[76349]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T01:34:29.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:31.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:31 smithi184 ceph-mon[100622]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:34:31.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:31 smithi184 ceph-mon[102844]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:34:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:31 smithi196 ceph-mon[76349]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T01:34:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:31.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:31.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:31.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:34:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[100622]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[102844]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:33 smithi196 ceph-mon[76349]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[102844]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[100622]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:35 smithi196 ceph-mon[76349]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:34:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:36 smithi184 ceph-mon[100622]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:34:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:36 smithi184 ceph-mon[102844]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:34:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:36 smithi196 ceph-mon[76349]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:34:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:39 smithi196 ceph-mon[76349]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T01:34:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:39 smithi184 ceph-mon[100622]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T01:34:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:39 smithi184 ceph-mon[102844]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T01:34:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:41 smithi184 ceph-mon[102844]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T01:34:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:41 smithi184 ceph-mon[100622]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T01:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:41.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:41.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:41.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:41 smithi196 ceph-mon[76349]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T01:34:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:34:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:43 smithi184 ceph-mon[102844]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:34:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:43 smithi184 ceph-mon[100622]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:34:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:43 smithi196 ceph-mon[76349]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:34:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:45 smithi184 ceph-mon[100622]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T01:34:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:45 smithi184 ceph-mon[102844]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T01:34:45.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:45 smithi196 ceph-mon[76349]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T01:34:47.008 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:34:47.535 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:34:47.535 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T01:34:47.535 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 5h - - 2024-09-21T01:34:47.535 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 12s ago 5h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 12s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 5h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:34:47.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 12s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 5h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:34:47.537 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 12s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:34:47.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:47 smithi196 ceph-mon[76349]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T01:34:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:47 smithi184 ceph-mon[100622]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T01:34:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:47 smithi184 ceph-mon[102844]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T01:34:47.849 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:34:47.849 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:34:47.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:34:47.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:34:47.850 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:34:48.119 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:34:48.120 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:34:48.433 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:34:48.433 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:48.433 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:48 smithi196 ceph-mon[76349]: from='client.67293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:48 smithi196 ceph-mon[76349]: from='client.77264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3558317076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/878221293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[100622]: from='client.67293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[100622]: from='client.77264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3558317076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/878221293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[102844]: from='client.67293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[102844]: from='client.77264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3558317076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/878221293' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:49.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:49 smithi196 ceph-mon[76349]: from='client.77270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:49 smithi196 ceph-mon[76349]: from='client.67317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:49 smithi196 ceph-mon[76349]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[100622]: from='client.77270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[100622]: from='client.67317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[100622]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[102844]: from='client.77270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[102844]: from='client.67317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:49 smithi184 ceph-mon[102844]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:34:51.748 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:51 smithi184 ceph-mon[100622]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T01:34:51.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:51 smithi184 ceph-mon[102844]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T01:34:51.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:51.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:51.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:51.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:34:51.748 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:34:51.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:34:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:51 smithi196 ceph-mon[76349]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T01:34:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:34:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:34:53.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:53 smithi184 ceph-mon[100622]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:53.593 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:53 smithi184 ceph-mon[102844]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:53 smithi196 ceph-mon[76349]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[100622]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[102844]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:55 smithi196 ceph-mon[76349]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:34:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:57 smithi184 ceph-mon[100622]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:57 smithi184 ceph-mon[102844]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:57 smithi196 ceph-mon[76349]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:59 smithi184 ceph-mon[100622]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:59 smithi184 ceph-mon[102844]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:59 smithi196 ceph-mon[76349]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:01 smithi184 ceph-mon[100622]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:01 smithi184 ceph-mon[102844]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:01.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:01.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:01.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:01 smithi196 ceph-mon[76349]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:35:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:03 smithi184 ceph-mon[100622]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:03 smithi184 ceph-mon[102844]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:03 smithi196 ceph-mon[76349]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:05 smithi184 ceph-mon[100622]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:05 smithi184 ceph-mon[102844]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:05 smithi196 ceph-mon[76349]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:06 smithi184 ceph-mon[102844]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:06 smithi184 ceph-mon[100622]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:06 smithi196 ceph-mon[76349]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:09 smithi196 ceph-mon[76349]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:09 smithi184 ceph-mon[100622]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:09 smithi184 ceph-mon[102844]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:11.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:11 smithi196 ceph-mon[76349]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:11 smithi184 ceph-mon[100622]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:11 smithi184 ceph-mon[102844]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:11.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:11.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:11.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:35:13.733 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:13 smithi184 ceph-mon[100622]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:13.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:13 smithi184 ceph-mon[102844]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:13 smithi196 ceph-mon[76349]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:14.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:14.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.767 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:15 smithi196 ceph-mon[76349]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[100622]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[102844]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:17 smithi184 ceph-mon[100622]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:17 smithi184 ceph-mon[102844]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:17 smithi196 ceph-mon[76349]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:18.728 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:35:19.252 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:35:19.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 5h - - 2024-09-21T01:35:19.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 5h - - 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 3s ago 5h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 5s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 3s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:35:19.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 3s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 3s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 5h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:35:19.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:35:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:19 smithi184 ceph-mon[100622]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:19 smithi184 ceph-mon[102844]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:35:19.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:35:19.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:19.563 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:35:19.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:35:19.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:35:19.563 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:35:19.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:35:19.834 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:35:19.835 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:35:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:19 smithi196 ceph-mon[76349]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:20.143 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:35:20.143 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:20.143 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:20.143 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:20.143 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[100622]: from='client.77294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[100622]: from='client.67335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[100622]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/167185973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1342364008' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[102844]: from='client.77294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[102844]: from='client.67335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[102844]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/167185973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1342364008' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:20 smithi196 ceph-mon[76349]: from='client.77294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:20 smithi196 ceph-mon[76349]: from='client.67335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:20 smithi196 ceph-mon[76349]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/167185973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1342364008' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-mon[100622]: from='client.67353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-mon[100622]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-mon[102844]: from='client.67353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-mon[102844]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:21.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:21.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:21.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:21 smithi196 ceph-mon[76349]: from='client.67353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:21 smithi196 ceph-mon[76349]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:35:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:23 smithi184 ceph-mon[100622]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:23 smithi184 ceph-mon[102844]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:23 smithi196 ceph-mon[76349]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:25 smithi184 ceph-mon[100622]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:25 smithi184 ceph-mon[102844]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:25 smithi196 ceph-mon[76349]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:27 smithi184 ceph-mon[100622]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:27 smithi184 ceph-mon[102844]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:27 smithi196 ceph-mon[76349]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:29 smithi196 ceph-mon[76349]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.764 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:29 smithi184 ceph-mon[100622]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:29 smithi184 ceph-mon[102844]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:31 smithi184 ceph-mon[100622]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:31 smithi184 ceph-mon[102844]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:31.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:31.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:31.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:31 smithi196 ceph-mon[76349]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:35:33.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:33 smithi184 ceph-mon[100622]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:33.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:33 smithi184 ceph-mon[102844]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:33 smithi196 ceph-mon[76349]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:34.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[100622]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[102844]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:35 smithi196 ceph-mon[76349]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:36 smithi184 ceph-mon[100622]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:36 smithi184 ceph-mon[102844]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:36 smithi196 ceph-mon[76349]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:39 smithi184 ceph-mon[100622]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:39 smithi184 ceph-mon[102844]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:39 smithi196 ceph-mon[76349]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:41 smithi184 ceph-mon[100622]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:41 smithi184 ceph-mon[102844]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:41.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:41.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:41.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:41 smithi196 ceph-mon[76349]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:35:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:43 smithi184 ceph-mon[100622]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:43 smithi184 ceph-mon[102844]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:43 smithi196 ceph-mon[76349]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:45 smithi184 ceph-mon[100622]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:45 smithi184 ceph-mon[102844]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:45.805 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:45 smithi196 ceph-mon[76349]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:47 smithi184 ceph-mon[100622]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:47 smithi184 ceph-mon[102844]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:47 smithi196 ceph-mon[76349]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:49.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:49 smithi196 ceph-mon[76349]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:49 smithi184 ceph-mon[100622]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:49 smithi184 ceph-mon[102844]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:50.439 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:35:50.964 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:35:50.964 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T01:35:50.964 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 5h - - 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 15s ago 5h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 5h 532M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 15s ago 5h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:35:50.965 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 5h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:35:50.966 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:35:51.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:35:51.274 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:35:51.544 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:35:51.544 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:35:51.545 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[100622]: from='client.77330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[100622]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2962165732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[102844]: from='client.77330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[102844]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:51.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2962165732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:51.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:51.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:35:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:35:51.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:35:51.859 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:35:51.859 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:51.859 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:51.859 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:51.859 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:51.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:51 smithi196 ceph-mon[76349]: from='client.77330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:51.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:51 smithi196 ceph-mon[76349]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:51.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2962165732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:35:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[100622]: from='client.77336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[100622]: from='client.77342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1102657613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[102844]: from='client.77336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[102844]: from='client.77342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1102657613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:52 smithi196 ceph-mon[76349]: from='client.77336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:52 smithi196 ceph-mon[76349]: from='client.77342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1102657613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:53 smithi184 ceph-mon[100622]: from='client.77354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:53 smithi184 ceph-mon[100622]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:53 smithi184 ceph-mon[102844]: from='client.77354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:53 smithi184 ceph-mon[102844]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:53 smithi196 ceph-mon[76349]: from='client.77354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:53 smithi196 ceph-mon[76349]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:55 smithi184 ceph-mon[100622]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:55 smithi184 ceph-mon[102844]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:55 smithi196 ceph-mon[76349]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[100622]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:56 smithi184 ceph-mon[102844]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:35:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:56 smithi196 ceph-mon[76349]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:59 smithi184 ceph-mon[100622]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:35:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:35:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:59 smithi184 ceph-mon[102844]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:35:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:35:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:59 smithi196 ceph-mon[76349]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:35:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:35:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:01 smithi196 ceph-mon[76349]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:01 smithi184 ceph-mon[100622]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:01 smithi184 ceph-mon[102844]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:01.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:01.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:01.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:36:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:03 smithi184 ceph-mon[100622]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:03 smithi184 ceph-mon[102844]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:03 smithi196 ceph-mon[76349]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:05 smithi184 ceph-mon[100622]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:05 smithi184 ceph-mon[102844]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:05 smithi196 ceph-mon[76349]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:07 smithi184 ceph-mon[100622]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:07 smithi184 ceph-mon[102844]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:07 smithi196 ceph-mon[76349]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:09 smithi184 ceph-mon[100622]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:36:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:09 smithi184 ceph-mon[102844]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:36:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:09 smithi196 ceph-mon[76349]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:36:11.607 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:11 smithi196 ceph-mon[76349]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:11 smithi184 ceph-mon[100622]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:11 smithi184 ceph-mon[102844]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:11.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:11.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:11.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:11.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:36:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:13 smithi184 ceph-mon[100622]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:13 smithi184 ceph-mon[102844]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:13 smithi196 ceph-mon[76349]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:15 smithi184 ceph-mon[100622]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:15.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:15 smithi184 ceph-mon[102844]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:15 smithi196 ceph-mon[76349]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:36:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[100622]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:16 smithi184 ceph-mon[102844]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:16 smithi196 ceph-mon[76349]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:19 smithi184 ceph-mon[102844]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:19 smithi184 ceph-mon[100622]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:19 smithi196 ceph-mon[76349]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:36:21.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:21 smithi184 ceph-mon[102844]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:21 smithi184 ceph-mon[100622]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:21.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:21.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:21.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:21 smithi196 ceph-mon[76349]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:22.157 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:36:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 5h - - 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 5h - - 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 8s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 6s ago 5h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:36:22.700 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 5h 532M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 8s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 6s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 6s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 6s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 6s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:36:22.701 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 5h 67.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:36:22.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:36:22.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:36:22.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:36:22.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:36:23.012 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:36:23.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:36:23.280 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:36:23.280 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:36:23.281 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:36:23.282 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:36:23.282 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:36:23.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[100622]: from='client.67401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[100622]: from='client.77372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[100622]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3697550412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[102844]: from='client.67401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[102844]: from='client.77372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[102844]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:23.558 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3697550412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:23.590 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:36:23.591 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:23.591 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:23.591 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:23 smithi196 ceph-mon[76349]: from='client.67401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:23 smithi196 ceph-mon[76349]: from='client.77372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:23 smithi196 ceph-mon[76349]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3697550412' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[100622]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[100622]: from='client.77390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2180048952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[102844]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[102844]: from='client.77390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2180048952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:24 smithi196 ceph-mon[76349]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:24 smithi196 ceph-mon[76349]: from='client.77390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2180048952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:25 smithi184 ceph-mon[100622]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:25 smithi184 ceph-mon[102844]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:25 smithi196 ceph-mon[76349]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:36:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:27 smithi184 ceph-mon[100622]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:27 smithi184 ceph-mon[102844]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:27 smithi196 ceph-mon[76349]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:29 smithi184 ceph-mon[100622]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:29 smithi184 ceph-mon[102844]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:29 smithi196 ceph-mon[76349]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:36:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:31 smithi184 ceph-mon[100622]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:31 smithi184 ceph-mon[102844]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:31.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:31.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:31.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:31.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:31.859 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:31 smithi196 ceph-mon[76349]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:36:33.695 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:33 smithi196 ceph-mon[76349]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:33 smithi184 ceph-mon[100622]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:33 smithi184 ceph-mon[102844]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:34.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:34.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:35 smithi184 ceph-mon[100622]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:35 smithi184 ceph-mon[102844]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:35.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:35 smithi196 ceph-mon[76349]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:37.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.029 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[100622]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:36 smithi184 ceph-mon[102844]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:36 smithi196 ceph-mon[76349]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:39 smithi184 ceph-mon[100622]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:39 smithi184 ceph-mon[102844]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:39 smithi196 ceph-mon[76349]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:41 smithi184 ceph-mon[100622]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:41 smithi184 ceph-mon[102844]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:41.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:41.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:41.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:41 smithi196 ceph-mon[76349]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:36:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:43 smithi184 ceph-mon[100622]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:43 smithi184 ceph-mon[102844]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:43 smithi196 ceph-mon[76349]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:45 smithi184 ceph-mon[100622]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:45 smithi184 ceph-mon[102844]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:45 smithi196 ceph-mon[76349]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:47 smithi184 ceph-mon[100622]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:47 smithi184 ceph-mon[102844]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:47 smithi196 ceph-mon[76349]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:49 smithi184 ceph-mon[100622]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:49 smithi184 ceph-mon[102844]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:49 smithi196 ceph-mon[76349]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:51.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:51 smithi196 ceph-mon[76349]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:51 smithi184 ceph-mon[100622]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:51.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:51 smithi184 ceph-mon[102844]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:51.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:51.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:51.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:36:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:36:51.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:36:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:36:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:36:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:52 smithi196 ceph-mon[76349]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:52 smithi184 ceph-mon[100622]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:52 smithi184 ceph-mon[102844]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:53.880 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:53.880 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:36:53.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 5h - - 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 18s ago 5h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 5h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 18s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 5h 68.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:54.730 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:36:54.731 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:36:55.001 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:36:55.312 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:36:55.312 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:55.313 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:55.313 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:55.313 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='client.77402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='client.77414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2033586317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/180203461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='client.77402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='client.77414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2033586317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/180203461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='client.77402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='client.77414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2033586317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/180203461' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:56 smithi196 ceph-mon[76349]: from='client.77426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[100622]: from='client.77426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:56.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[102844]: from='client.77426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:56.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:36:57.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:57 smithi184 ceph-mon[100622]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:57.651 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:57 smithi184 ceph-mon[102844]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:57 smithi196 ceph-mon[76349]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:59 smithi184 ceph-mon[102844]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:36:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:59 smithi184 ceph-mon[100622]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:36:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:59 smithi196 ceph-mon[76349]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:36:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:01 smithi184 ceph-mon[100622]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:01.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:01 smithi184 ceph-mon[102844]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:01.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:01.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:01.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:01 smithi196 ceph-mon[76349]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:37:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:03 smithi184 ceph-mon[100622]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:03 smithi184 ceph-mon[102844]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:03 smithi196 ceph-mon[76349]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:05 smithi184 ceph-mon[100622]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:05 smithi184 ceph-mon[102844]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:05 smithi196 ceph-mon[76349]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:07 smithi184 ceph-mon[100622]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:07.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:07 smithi184 ceph-mon[102844]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:07 smithi196 ceph-mon[76349]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:08 smithi196 ceph-mon[76349]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:08 smithi184 ceph-mon[100622]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:08 smithi184 ceph-mon[102844]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:11.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:11 smithi184 ceph-mon[100622]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:11.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:11 smithi184 ceph-mon[102844]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:11.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded" 2024-09-21T01:37:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:11.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:11.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:11 smithi196 ceph-mon[76349]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:37:13.636 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:13 smithi196 ceph-mon[76349]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:37:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:13 smithi184 ceph-mon[100622]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:37:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:13 smithi184 ceph-mon[102844]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:15 smithi196 ceph-mon[76349]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:15 smithi184 ceph-mon[100622]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:15 smithi184 ceph-mon[102844]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:17 smithi196 ceph-mon[76349]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:17.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[100622]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:17.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:17 smithi184 ceph-mon[102844]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:19.621 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:19 smithi184 ceph-mon[100622]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:19.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:19 smithi184 ceph-mon[102844]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:19 smithi196 ceph-mon[76349]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:21 smithi184 ceph-mon[100622]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:21 smithi184 ceph-mon[102844]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:21.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:21.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:21.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:21.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:21 smithi196 ceph-mon[76349]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:37:23.803 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:23 smithi184 ceph-mon[100622]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:23.803 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:23 smithi184 ceph-mon[102844]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:23 smithi196 ceph-mon[76349]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:25.606 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:37:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:25 smithi184 ceph-mon[100622]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:25 smithi184 ceph-mon[102844]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:25 smithi196 ceph-mon[76349]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 5h - - 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 11s ago 5h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 92.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 11s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:37:26.132 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 11s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 11s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 10s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 11s ago 5h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 11s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 11s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 11s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 11s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 11s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 10s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 10s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 10s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 10s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 10s ago 5h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:37:26.133 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:37:26.134 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:37:26.134 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:37:26.134 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 10s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:37:26.442 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:37:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:37:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:37:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:37:26.444 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:37:26.712 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:37:26.712 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:37:26.712 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:37:26.713 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[100622]: from='client.77438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[100622]: from='client.77444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[100622]: from='client.67485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2003203004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[100622]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[102844]: from='client.77438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[102844]: from='client.77444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[102844]: from='client.67485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2003203004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:26.715 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:26 smithi184 ceph-mon[102844]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:26 smithi196 ceph-mon[76349]: from='client.77438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:26 smithi196 ceph-mon[76349]: from='client.77444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:26 smithi196 ceph-mon[76349]: from='client.67485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2003203004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:26 smithi196 ceph-mon[76349]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:27.022 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:37:27.022 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:27.023 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:27.023 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:27.023 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:27 smithi184 ceph-mon[100622]: from='client.67497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1009778451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:27 smithi184 ceph-mon[102844]: from='client.67497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1009778451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:27 smithi196 ceph-mon[76349]: from='client.67497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1009778451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:28 smithi184 ceph-mon[100622]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:28 smithi184 ceph-mon[102844]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:28 smithi196 ceph-mon[76349]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:30 smithi184 ceph-mon[100622]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:30 smithi184 ceph-mon[102844]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:30 smithi196 ceph-mon[76349]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:31.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:31.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:31.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:37:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:33 smithi184 ceph-mon[100622]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:33 smithi184 ceph-mon[102844]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:33.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:33 smithi196 ceph-mon[76349]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[100622]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[102844]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:35 smithi196 ceph-mon[76349]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[100622]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:37 smithi184 ceph-mon[102844]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:37 smithi196 ceph-mon[76349]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:39 smithi184 ceph-mon[100622]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:39.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:39 smithi184 ceph-mon[102844]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:39 smithi196 ceph-mon[76349]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:41 smithi184 ceph-mon[100622]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:41.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:41 smithi184 ceph-mon[102844]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:41.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:41.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:41.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:41.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:41 smithi196 ceph-mon[76349]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:37:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:43 smithi184 ceph-mon[100622]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:43 smithi184 ceph-mon[102844]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:43 smithi196 ceph-mon[76349]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:44.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:45 smithi184 ceph-mon[100622]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:45 smithi184 ceph-mon[102844]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:45.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:45 smithi196 ceph-mon[76349]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:46 smithi184 ceph-mon[100622]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:46 smithi184 ceph-mon[102844]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:46 smithi196 ceph-mon[76349]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:49.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:49 smithi184 ceph-mon[100622]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:49.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:49 smithi184 ceph-mon[102844]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:49 smithi196 ceph-mon[76349]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:51 smithi184 ceph-mon[100622]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:51 smithi184 ceph-mon[102844]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:51.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:51.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:37:51.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:37:51.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:37:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:51 smithi196 ceph-mon[76349]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:37:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:37:53.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:53 smithi196 ceph-mon[76349]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:53 smithi184 ceph-mon[100622]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:53 smithi184 ceph-mon[102844]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:54.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:54 smithi196 ceph-mon[76349]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:54.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:54.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:54 smithi184 ceph-mon[100622]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:54.890 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:54.890 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:54 smithi184 ceph-mon[102844]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.316 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:37:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:57 smithi196 ceph-mon[76349]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[100622]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:37:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:57 smithi184 ceph-mon[102844]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:57.847 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 5h - - 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 1s ago 5h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 1s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:37:57.848 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 1s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 1s ago 5h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:37:57.849 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:37:57.850 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:37:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:37:58.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:37:58.430 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:37:58.430 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:37:58.430 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:37:58.430 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:37:58.430 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:37:58.431 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:37:58.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:58 smithi184 ceph-mon[100622]: from='client.77474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:58.431 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/612290897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:58.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:58 smithi184 ceph-mon[102844]: from='client.77474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:58.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/612290897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:58.747 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:37:58.747 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:58.747 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:58.747 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:58.747 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:58 smithi196 ceph-mon[76349]: from='client.77474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/612290897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:59.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: from='client.67515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: from='client.67521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: from='client.67533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.682 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1229293027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: from='client.67515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: from='client.67521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: from='client.67533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1229293027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:59.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: from='client.67515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: from='client.67521 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: from='client.67533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1229293027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:01.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:01 smithi184 ceph-mon[100622]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:01 smithi184 ceph-mon[102844]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:01.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:01.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:01.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:01.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:01 smithi196 ceph-mon[76349]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:38:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:03 smithi184 ceph-mon[100622]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:03 smithi184 ceph-mon[102844]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:03 smithi196 ceph-mon[76349]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:05 smithi184 ceph-mon[100622]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:05 smithi184 ceph-mon[102844]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:05 smithi196 ceph-mon[76349]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:06 smithi184 ceph-mon[100622]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:06 smithi184 ceph-mon[102844]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:06 smithi196 ceph-mon[76349]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:09.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:09 smithi184 ceph-mon[100622]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:09.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:09 smithi184 ceph-mon[102844]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:09 smithi196 ceph-mon[76349]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:10 smithi196 ceph-mon[76349]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:10 smithi184 ceph-mon[100622]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:10 smithi184 ceph-mon[102844]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:11.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:11.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:11.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:38:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:13 smithi184 ceph-mon[102844]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:13 smithi184 ceph-mon[100622]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:13.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:13 smithi196 ceph-mon[76349]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:38:13.812452046Z level=info msg="Completed cleanup jobs" duration=12.985252ms 2024-09-21T01:38:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:15 smithi196 ceph-mon[76349]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[102844]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[100622]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[100622]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[102844]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:17 smithi196 ceph-mon[76349]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:19.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:19 smithi184 ceph-mon[100622]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:19.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:19 smithi184 ceph-mon[102844]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:19 smithi196 ceph-mon[76349]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:21 smithi184 ceph-mon[100622]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:21 smithi184 ceph-mon[102844]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:21.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:21.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:21.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:21.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:21 smithi196 ceph-mon[76349]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:38:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:23 smithi184 ceph-mon[100622]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:23 smithi184 ceph-mon[102844]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:23 smithi196 ceph-mon[76349]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:24.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:25 smithi184 ceph-mon[100622]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:25 smithi184 ceph-mon[102844]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:25 smithi196 ceph-mon[76349]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:27 smithi184 ceph-mon[100622]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:27 smithi184 ceph-mon[102844]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:27 smithi196 ceph-mon[76349]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:29.042 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:38:29.570 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:38:29.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 5h - - 2024-09-21T01:38:29.570 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T01:38:29.570 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 14s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 92.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:38:29.571 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 5h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 14s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:38:29.572 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:38:29.573 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:38:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:29 smithi184 ceph-mon[102844]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:29 smithi184 ceph-mon[100622]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:38:29.883 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:38:29.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:38:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:29 smithi196 ceph-mon[76349]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:30.154 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:38:30.154 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:38:30.154 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:38:30.154 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:38:30.154 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:38:30.155 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[102844]: from='client.67545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[102844]: from='client.77516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3322516912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/325861118' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[100622]: from='client.67545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[100622]: from='client.77516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3322516912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/325861118' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:30 smithi196 ceph-mon[76349]: from='client.67545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:30 smithi196 ceph-mon[76349]: from='client.77516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3322516912' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/325861118' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[100622]: from='client.77522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[100622]: from='client.67569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[100622]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:31.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[102844]: from='client.77522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[102844]: from='client.67569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-mon[102844]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:31.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:31.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:31.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:31.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:31.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:31 smithi196 ceph-mon[76349]: from='client.77522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:31 smithi196 ceph-mon[76349]: from='client.67569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:31 smithi196 ceph-mon[76349]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:38:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:32 smithi196 ceph-mon[76349]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:32 smithi184 ceph-mon[100622]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:32 smithi184 ceph-mon[102844]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:34 smithi196 ceph-mon[76349]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:35.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:34 smithi184 ceph-mon[100622]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:35.030 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:34 smithi184 ceph-mon[102844]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:36.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[102844]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[100622]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:37 smithi196 ceph-mon[76349]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:38.875 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:38 smithi196 ceph-mon[76349]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:39.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:38 smithi184 ceph-mon[100622]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:39.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:38 smithi184 ceph-mon[102844]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:40 smithi196 ceph-mon[76349]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:40 smithi184 ceph-mon[100622]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:40 smithi184 ceph-mon[102844]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:41.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:41.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:41.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:41.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:42.288 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:38:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:42 smithi196 ceph-mon[76349]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:42.946 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:42 smithi184 ceph-mon[100622]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:42.946 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:42 smithi184 ceph-mon[102844]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:44.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:44.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:44 smithi196 ceph-mon[76349]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:44 smithi184 ceph-mon[102844]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:44 smithi184 ceph-mon[100622]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:46 smithi196 ceph-mon[76349]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:46 smithi184 ceph-mon[100622]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:46 smithi184 ceph-mon[102844]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:48 smithi196 ceph-mon[76349]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:48 smithi184 ceph-mon[100622]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:48 smithi184 ceph-mon[102844]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:50 smithi196 ceph-mon[76349]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:50 smithi184 ceph-mon[100622]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:50 smithi184 ceph-mon[102844]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:51.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:51.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:38:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:38:51.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:38:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:38:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:38:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:52 smithi196 ceph-mon[76349]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:52.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:52 smithi184 ceph-mon[100622]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:52.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:52 smithi184 ceph-mon[102844]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:54 smithi196 ceph-mon[76349]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:54 smithi184 ceph-mon[100622]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:54 smithi184 ceph-mon[102844]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:57 smithi196 ceph-mon[76349]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[102844]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[100622]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:38:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:58 smithi196 ceph-mon[76349]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:58 smithi184 ceph-mon[100622]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:58 smithi184 ceph-mon[102844]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:00.768 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:39:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:00 smithi196 ceph-mon[76349]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:00.894 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:00 smithi184 ceph-mon[100622]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:00.894 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:00 smithi184 ceph-mon[102844]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 5h - - 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 4s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 4s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 6s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 6s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:39:01.309 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 4s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 6s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 18.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 4s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 4s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 4s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 4s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 4s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 4s ago 5h 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:39:01.310 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:39:01.311 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:39:01.311 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 4s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:39:01.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:39:01.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:01.627 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:39:01.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:39:01.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:39:01.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:39:01.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[100622]: from='client.77546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[100622]: from='client.77552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.874 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[100622]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.875 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[102844]: from='client.77546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.875 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[102844]: from='client.77552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.875 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-mon[102844]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:01.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:01.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:01.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:01.875 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:01.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:01 smithi196 ceph-mon[76349]: from='client.77546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:01 smithi196 ceph-mon[76349]: from='client.77552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:01 smithi196 ceph-mon[76349]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:39:01.906 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:39:02.160 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:39:02.230 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:39:02.230 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:02.230 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:02.230 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:02.230 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:02.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2295540161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[100622]: from='client.67605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/93244965' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[100622]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2295540161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[102844]: from='client.67605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/93244965' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:02.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:02 smithi184 ceph-mon[102844]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2295540161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:02 smithi196 ceph-mon[76349]: from='client.67605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/93244965' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:02 smithi196 ceph-mon[76349]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:04 smithi184 ceph-mon[100622]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:04 smithi184 ceph-mon[102844]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:04 smithi196 ceph-mon[76349]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:06 smithi196 ceph-mon[76349]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:06 smithi184 ceph-mon[100622]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:06 smithi184 ceph-mon[102844]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:08 smithi196 ceph-mon[76349]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:08 smithi184 ceph-mon[100622]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:08 smithi184 ceph-mon[102844]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:10 smithi196 ceph-mon[76349]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:10 smithi184 ceph-mon[100622]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:10 smithi184 ceph-mon[102844]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:11.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:11.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:11.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:39:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:12 smithi196 ceph-mon[76349]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:12 smithi184 ceph-mon[100622]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:12 smithi184 ceph-mon[102844]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:14 smithi196 ceph-mon[76349]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:14 smithi184 ceph-mon[102844]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:14 smithi184 ceph-mon[100622]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:16.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[100622]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[102844]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:17 smithi196 ceph-mon[76349]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:17.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:17.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:18.723 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:18 smithi196 ceph-mon[76349]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:18 smithi184 ceph-mon[100622]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:18 smithi184 ceph-mon[102844]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:20 smithi196 ceph-mon[76349]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:20 smithi184 ceph-mon[100622]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:20 smithi184 ceph-mon[102844]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:21.742 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:21.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:21.742 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:21.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:21.742 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:21.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:39:22.787 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:22 smithi184 ceph-mon[100622]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:39:22.787 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:22 smithi184 ceph-mon[102844]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:39:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:22 smithi196 ceph-mon[76349]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:39:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:24.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:24.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:24 smithi196 ceph-mon[76349]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:24 smithi184 ceph-mon[100622]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:24 smithi184 ceph-mon[102844]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:26 smithi196 ceph-mon[76349]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:26 smithi184 ceph-mon[100622]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:26 smithi184 ceph-mon[102844]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:28 smithi196 ceph-mon[76349]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:28 smithi184 ceph-mon[100622]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:28 smithi184 ceph-mon[102844]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:30 smithi196 ceph-mon[76349]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:30 smithi184 ceph-mon[100622]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:30 smithi184 ceph-mon[102844]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:31.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:31.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:31.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:39:32.528 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:39:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:32 smithi196 ceph-mon[76349]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:32 smithi196 ceph-mon[76349]: from='client.77582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:32 smithi184 ceph-mon[102844]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:33.092 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:32 smithi184 ceph-mon[102844]: from='client.77582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.093 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:32 smithi184 ceph-mon[100622]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:33.093 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:32 smithi184 ceph-mon[100622]: from='client.77582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 5h - - 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 17s ago 5h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:39:33.093 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 16s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 17s ago 5h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:39:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 16s ago 5h 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:39:33.095 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:39:33.095 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:39:33.095 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:39:33.095 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:39:33.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:39:33.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:39:33.640 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:39:33.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[102844]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[102844]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:33.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2039578932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:33.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[100622]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[100622]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:33.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2039578932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:33 smithi196 ceph-mon[76349]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:33 smithi196 ceph-mon[76349]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:33.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2039578932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:33.955 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:39:33.956 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:33.956 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:33.956 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:33.956 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:34 smithi196 ceph-mon[76349]: from='client.77606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2392799546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:34 smithi196 ceph-mon[76349]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[100622]: from='client.77606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2392799546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[100622]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[102844]: from='client.77606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2392799546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:34 smithi184 ceph-mon[102844]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:36.853 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[102844]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[100622]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:37 smithi196 ceph-mon[76349]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:38 smithi196 ceph-mon[76349]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:38 smithi184 ceph-mon[102844]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:38 smithi184 ceph-mon[100622]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:40 smithi196 ceph-mon[76349]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:40 smithi184 ceph-mon[102844]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:40 smithi184 ceph-mon[100622]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:41.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:41.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:41.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:42.278 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:39:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:42 smithi196 ceph-mon[76349]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:42 smithi184 ceph-mon[100622]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:42 smithi184 ceph-mon[102844]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:44 smithi184 ceph-mon[102844]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:44 smithi184 ceph-mon[100622]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:44 smithi196 ceph-mon[76349]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:46 smithi196 ceph-mon[76349]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:46 smithi184 ceph-mon[100622]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:46 smithi184 ceph-mon[102844]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:48 smithi196 ceph-mon[76349]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:48 smithi184 ceph-mon[102844]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:48 smithi184 ceph-mon[100622]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:50 smithi196 ceph-mon[76349]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:50 smithi184 ceph-mon[102844]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:50 smithi184 ceph-mon[100622]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:51.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:51.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:39:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:39:51.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:39:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:39:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:39:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:52 smithi196 ceph-mon[76349]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:52 smithi184 ceph-mon[100622]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:52 smithi184 ceph-mon[102844]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:54 smithi196 ceph-mon[76349]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:54 smithi184 ceph-mon[100622]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:54 smithi184 ceph-mon[102844]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[102844]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[100622]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:57 smithi196 ceph-mon[76349]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:39:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:59 smithi184 ceph-mon[100622]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:59 smithi184 ceph-mon[102844]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:59 smithi196 ceph-mon[76349]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:39:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:01 smithi196 ceph-mon[76349]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:01 smithi184 ceph-mon[100622]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:01 smithi184 ceph-mon[102844]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:01.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:01.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:01.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:40:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:02 smithi196 ceph-mon[76349]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:02.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:02 smithi184 ceph-mon[100622]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:02.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:02 smithi184 ceph-mon[102844]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:03.822 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:04.253 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:40:04.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[100622]: from='client.77618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:04.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[100622]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:04.817 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[100622]: from='client.77624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:04.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[102844]: from='client.77618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:04.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[102844]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:04.818 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:04 smithi184 ceph-mon[102844]: from='client.77624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 5h - - 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 5h - - 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 7s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:40:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 7s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 9s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 7s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 9s ago 5h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 7s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 7s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 7s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:40:04.819 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 7s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 7s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 7s ago 5h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:40:04.820 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 7s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:40:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:04 smithi196 ceph-mon[76349]: from='client.77618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:04 smithi196 ceph-mon[76349]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:04 smithi196 ceph-mon[76349]: from='client.77624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:05.134 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:40:05.135 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:40:05.432 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:40:05.433 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:40:05.747 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:05.747 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:05.747 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:05.747 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:05.747 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:05.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[100622]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2359511010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:05.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[100622]: from='client.67677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[102844]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2359511010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:05.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:05 smithi184 ceph-mon[102844]: from='client.67677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:05 smithi196 ceph-mon[76349]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2359511010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:05 smithi196 ceph-mon[76349]: from='client.67677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2066701062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:06 smithi196 ceph-mon[76349]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2066701062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:06 smithi184 ceph-mon[100622]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2066701062' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:06 smithi184 ceph-mon[102844]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:08 smithi196 ceph-mon[76349]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:08 smithi184 ceph-mon[100622]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:08 smithi184 ceph-mon[102844]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:10 smithi196 ceph-mon[76349]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:10 smithi184 ceph-mon[100622]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:10 smithi184 ceph-mon[102844]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:11.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:11.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:11.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:40:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:12 smithi196 ceph-mon[76349]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:12 smithi184 ceph-mon[100622]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:12 smithi184 ceph-mon[102844]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:14 smithi196 ceph-mon[76349]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:14 smithi184 ceph-mon[100622]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:14 smithi184 ceph-mon[102844]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[100622]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:16 smithi184 ceph-mon[102844]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:16 smithi196 ceph-mon[76349]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:18.376 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:19.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:19 smithi196 ceph-mon[76349]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:19 smithi184 ceph-mon[100622]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:19 smithi184 ceph-mon[102844]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:20.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:20 smithi196 ceph-mon[76349]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:20 smithi184 ceph-mon[100622]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:20 smithi184 ceph-mon[102844]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:21.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:21.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:21.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:40:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:22 smithi196 ceph-mon[76349]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:22 smithi184 ceph-mon[100622]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:22 smithi184 ceph-mon[102844]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:23.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:23.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:24 smithi196 ceph-mon[76349]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:24 smithi184 ceph-mon[100622]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:24 smithi184 ceph-mon[102844]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:26 smithi196 ceph-mon[76349]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:26 smithi184 ceph-mon[100622]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:26 smithi184 ceph-mon[102844]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:28 smithi196 ceph-mon[76349]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:28 smithi184 ceph-mon[100622]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:28 smithi184 ceph-mon[102844]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:30 smithi196 ceph-mon[76349]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:30 smithi184 ceph-mon[100622]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:30 smithi184 ceph-mon[102844]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:31.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:31.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:31.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:40:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:32 smithi196 ceph-mon[76349]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:32 smithi184 ceph-mon[100622]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:32 smithi184 ceph-mon[102844]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:34 smithi196 ceph-mon[76349]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:34 smithi184 ceph-mon[102844]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:34 smithi184 ceph-mon[100622]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:36.036 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 5h - - 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 5h - - 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 0s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 19s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 5h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:40:36.566 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 0s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:40:36.567 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:40:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:40:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:40:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:40:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:40:36.882 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: from='client.77654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: from='client.77660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:36.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[102844]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: from='client.77654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: from='client.77660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:36.883 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:36 smithi184 ceph-mon[100622]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: from='client.77654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: from='client.77660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:36 smithi196 ceph-mon[76349]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:40:37.150 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:40:37.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:40:37.466 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:37.467 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:37.467 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:37.467 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:37.467 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='client.77666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1623954143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='client.67713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4234218307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='client.77666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1623954143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='client.67713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4234218307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='client.77666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1623954143' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='client.67713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4234218307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:38 smithi196 ceph-mon[76349]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:38 smithi184 ceph-mon[100622]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:38 smithi184 ceph-mon[102844]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:40 smithi196 ceph-mon[76349]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:40 smithi184 ceph-mon[100622]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:40 smithi184 ceph-mon[102844]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:41.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:41.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:41.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:40:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:42 smithi196 ceph-mon[76349]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:42.992 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:42 smithi184 ceph-mon[100622]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:42.992 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:42 smithi184 ceph-mon[102844]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:43.835 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:43.835 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:44.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:44.882 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:44 smithi184 ceph-mon[100622]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:44.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:44.882 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:44 smithi184 ceph-mon[102844]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:44 smithi196 ceph-mon[76349]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:46.796 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:46 smithi184 ceph-mon[100622]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:46 smithi184 ceph-mon[102844]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:46 smithi196 ceph-mon[76349]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:48 smithi196 ceph-mon[76349]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:48 smithi184 ceph-mon[100622]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:48 smithi184 ceph-mon[102844]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:50 smithi196 ceph-mon[76349]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:50 smithi184 ceph-mon[100622]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:50 smithi184 ceph-mon[102844]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:51.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:51.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:40:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:40:51.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:40:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:40:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:40:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:52 smithi196 ceph-mon[76349]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:52 smithi184 ceph-mon[100622]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:52 smithi184 ceph-mon[102844]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:54 smithi196 ceph-mon[76349]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:54 smithi184 ceph-mon[102844]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:54 smithi184 ceph-mon[100622]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[100622]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:56 smithi184 ceph-mon[102844]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:57.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:56 smithi196 ceph-mon[76349]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:40:59.824 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:59 smithi196 ceph-mon[76349]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:59.824 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:59 smithi184 ceph-mon[100622]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:00.125 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:00.125 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:59 smithi184 ceph-mon[102844]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:00.125 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:00.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:00 smithi196 ceph-mon[76349]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:00 smithi184 ceph-mon[100622]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:00 smithi184 ceph-mon[102844]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:01.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:01.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:01.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:41:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:02 smithi196 ceph-mon[76349]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:02 smithi184 ceph-mon[100622]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:02 smithi184 ceph-mon[102844]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:04.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:04 smithi184 ceph-mon[100622]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:04.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:04 smithi184 ceph-mon[102844]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:04 smithi196 ceph-mon[76349]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:06 smithi184 ceph-mon[100622]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:06 smithi184 ceph-mon[102844]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:06 smithi196 ceph-mon[76349]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:07.749 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 5h - - 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 5h - - 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 10s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:41:08.276 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 10s ago 5h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 10s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 10s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 10s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 10s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 10s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 10s ago 5h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:41:08.277 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:41:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:41:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:41:08.278 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 10s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:41:08.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:41:08.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:08.589 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:41:08.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:41:08.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:41:08.589 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:41:08.859 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:41:08.860 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[100622]: from='client.67725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[100622]: from='client.77696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[100622]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[100622]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4199761629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[102844]: from='client.67725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[102844]: from='client.77696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[102844]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[102844]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:08.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4199761629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:08 smithi196 ceph-mon[76349]: from='client.67725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:08 smithi196 ceph-mon[76349]: from='client.77696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:08 smithi196 ceph-mon[76349]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:08 smithi196 ceph-mon[76349]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4199761629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:09.169 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:41:09.170 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:09.170 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:09.170 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:09.170 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:09 smithi196 ceph-mon[76349]: from='client.77714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1906145885' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:09 smithi184 ceph-mon[102844]: from='client.77714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1906145885' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:09 smithi184 ceph-mon[100622]: from='client.77714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1906145885' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:10 smithi196 ceph-mon[76349]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:10 smithi184 ceph-mon[102844]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:10 smithi184 ceph-mon[100622]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:11.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:11.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:11.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:41:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:12 smithi196 ceph-mon[76349]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:12 smithi184 ceph-mon[100622]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:12 smithi184 ceph-mon[102844]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:13.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:13.987 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:14 smithi196 ceph-mon[76349]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:14 smithi184 ceph-mon[100622]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:14 smithi184 ceph-mon[102844]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:17 smithi196 ceph-mon[76349]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[100622]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:17 smithi184 ceph-mon[102844]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[100622]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:18 smithi184 ceph-mon[102844]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:19.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:18 smithi196 ceph-mon[76349]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:20.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:20 smithi196 ceph-mon[76349]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:20 smithi184 ceph-mon[100622]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:20 smithi184 ceph-mon[102844]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:21.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:21.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:21.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:41:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:22 smithi196 ceph-mon[76349]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:22 smithi184 ceph-mon[102844]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:22 smithi184 ceph-mon[100622]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:24 smithi196 ceph-mon[76349]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:24 smithi184 ceph-mon[100622]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:24 smithi184 ceph-mon[102844]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:26 smithi196 ceph-mon[76349]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:26 smithi184 ceph-mon[100622]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:26 smithi184 ceph-mon[102844]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:28 smithi196 ceph-mon[76349]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:28 smithi184 ceph-mon[100622]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:28 smithi184 ceph-mon[102844]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:30 smithi196 ceph-mon[76349]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:30 smithi184 ceph-mon[100622]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:30 smithi184 ceph-mon[102844]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:31.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:31.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:31.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:41:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:32 smithi196 ceph-mon[76349]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:32 smithi184 ceph-mon[100622]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:32 smithi184 ceph-mon[102844]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:34 smithi196 ceph-mon[76349]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:34 smithi184 ceph-mon[100622]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:34 smithi184 ceph-mon[102844]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[100622]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:37 smithi184 ceph-mon[102844]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:37.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:37 smithi196 ceph-mon[76349]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:38 smithi196 ceph-mon[76349]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[102844]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:38 smithi184 ceph-mon[100622]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.466 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:41:39.993 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 5h - - 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 5h - - 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 3s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 3s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 3s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 3s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:41:39.994 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:41:39.995 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:41:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:39 smithi196 ceph-mon[76349]: from='client.77726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:39 smithi184 ceph-mon[102844]: from='client.77726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:39 smithi184 ceph-mon[100622]: from='client.77726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:40.305 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:41:40.306 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:41:40.576 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:41:40.577 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:41:40.577 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:41:40.577 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:41:40.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[100622]: from='client.77732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:40.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[100622]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:40.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2558686134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:40.828 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[100622]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:40.888 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:41:40.888 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:40.889 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:40.889 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:40.889 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:41.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:40 smithi196 ceph-mon[76349]: from='client.77732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:41.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:40 smithi196 ceph-mon[76349]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:41.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2558686134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:41.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:40 smithi196 ceph-mon[76349]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[102844]: from='client.77732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[102844]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2558686134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:40 smithi184 ceph-mon[102844]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:42.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-mon[102844]: from='client.67785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:42.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2486914459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-mon[100622]: from='client.67785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2486914459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:41.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:41.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:41.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:42.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:41 smithi196 ceph-mon[76349]: from='client.67785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:42.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2486914459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:41:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:42 smithi196 ceph-mon[76349]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:42 smithi184 ceph-mon[100622]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:42 smithi184 ceph-mon[102844]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:44 smithi196 ceph-mon[76349]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:44 smithi184 ceph-mon[100622]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:44 smithi184 ceph-mon[102844]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:46 smithi196 ceph-mon[76349]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:46.916 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:46 smithi184 ceph-mon[100622]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:46.917 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:46 smithi184 ceph-mon[102844]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:48 smithi184 ceph-mon[102844]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:48 smithi184 ceph-mon[100622]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:48 smithi196 ceph-mon[76349]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:50 smithi196 ceph-mon[76349]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:50 smithi184 ceph-mon[100622]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:50 smithi184 ceph-mon[102844]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:51.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:51.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:41:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:41:51.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:41:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:41:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:41:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:52 smithi196 ceph-mon[76349]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:52 smithi184 ceph-mon[100622]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:52 smithi184 ceph-mon[102844]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:54 smithi196 ceph-mon[76349]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:54 smithi184 ceph-mon[100622]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:54 smithi184 ceph-mon[102844]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:57 smithi196 ceph-mon[76349]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[100622]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:57 smithi184 ceph-mon[102844]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[100622]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:58 smithi184 ceph-mon[102844]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:59.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:58 smithi196 ceph-mon[76349]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:01.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:00 smithi196 ceph-mon[76349]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:00 smithi184 ceph-mon[102844]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:00 smithi184 ceph-mon[100622]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:01.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:01.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:01.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:42:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:02 smithi184 ceph-mon[100622]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:02 smithi184 ceph-mon[102844]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:02.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:02 smithi196 ceph-mon[76349]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:03.760 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:04 smithi196 ceph-mon[76349]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:04 smithi184 ceph-mon[100622]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:04 smithi184 ceph-mon[102844]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:06 smithi184 ceph-mon[100622]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:06 smithi184 ceph-mon[102844]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:06 smithi196 ceph-mon[76349]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:08 smithi196 ceph-mon[76349]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:08.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:08 smithi184 ceph-mon[100622]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:08.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:08 smithi184 ceph-mon[102844]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:10 smithi196 ceph-mon[76349]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:10 smithi184 ceph-mon[100622]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:10 smithi184 ceph-mon[102844]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:11.185 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:42:11.716 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 5h - - 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 5h - - 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 13s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 13s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 5h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:42:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 13s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 13s ago 5h 72.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:42:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:42:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:42:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:42:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:42:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:11 smithi196 ceph-mon[76349]: from='client.77762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:11 smithi196 ceph-mon[76349]: from='client.67803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-mon[102844]: from='client.77762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-mon[102844]: from='client.67803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-mon[100622]: from='client.77762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-mon[100622]: from='client.67803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:11.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:11.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:11.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:42:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:42:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:42:12.033 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:42:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:42:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:42:12.302 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:42:12.610 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:42:12.610 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:12.610 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:12.610 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:12.610 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:12 smithi196 ceph-mon[76349]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/419387305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:12 smithi196 ceph-mon[76349]: from='client.67821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:12 smithi196 ceph-mon[76349]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/288344739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[100622]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/419387305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[100622]: from='client.67821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[100622]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/288344739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[102844]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/419387305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[102844]: from='client.67821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[102844]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/288344739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:14.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:14 smithi196 ceph-mon[76349]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:14 smithi184 ceph-mon[100622]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:14 smithi184 ceph-mon[102844]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[100622]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:17.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:17 smithi184 ceph-mon[102844]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:17 smithi196 ceph-mon[76349]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:19 smithi196 ceph-mon[76349]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[100622]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:19 smithi184 ceph-mon[102844]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:20 smithi196 ceph-mon[76349]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:20 smithi184 ceph-mon[100622]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:20 smithi184 ceph-mon[102844]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:21.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:21.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:21.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:42:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:22 smithi196 ceph-mon[76349]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:22 smithi184 ceph-mon[100622]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:22 smithi184 ceph-mon[102844]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:23.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:23 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:23 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:23 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:24 smithi196 ceph-mon[76349]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:24 smithi184 ceph-mon[102844]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:24 smithi184 ceph-mon[100622]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:26 smithi196 ceph-mon[76349]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:26 smithi184 ceph-mon[100622]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:26 smithi184 ceph-mon[102844]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:28 smithi184 ceph-mon[100622]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:28 smithi184 ceph-mon[102844]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:28 smithi196 ceph-mon[76349]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:30 smithi196 ceph-mon[76349]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:30 smithi184 ceph-mon[100622]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:30 smithi184 ceph-mon[102844]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:31.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:31.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:31.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:42:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:32 smithi196 ceph-mon[76349]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:32 smithi184 ceph-mon[100622]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:32 smithi184 ceph-mon[102844]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:33 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:33 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:33 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:34 smithi184 ceph-mon[100622]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:34 smithi184 ceph-mon[102844]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:34 smithi196 ceph-mon[76349]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:37.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:37 smithi196 ceph-mon[76349]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:37.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[100622]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[102844]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[100622]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:39 smithi184 ceph-mon[102844]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:39 smithi196 ceph-mon[76349]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:40 smithi196 ceph-mon[76349]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:40 smithi184 ceph-mon[100622]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:40 smithi184 ceph-mon[102844]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:41.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:41.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:41.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:42:42.778 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:42 smithi196 ceph-mon[76349]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:42.903 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:42 smithi184 ceph-mon[100622]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:42.904 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:42 smithi184 ceph-mon[102844]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:42.904 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:42:43.425 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:42:43.425 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 5h - - 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 5h - - 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 6s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 6s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 6s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 6s ago 6h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 6s ago 5h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 6s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:42:43.426 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 72.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:42:43.427 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:42:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:42:43.735 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:42:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[100622]: from='client.77798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[100622]: from='client.77804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:43.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[100622]: from='client.77810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[102844]: from='client.77798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[102844]: from='client.77804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:43.736 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:43 smithi184 ceph-mon[102844]: from='client.77810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:43 smithi196 ceph-mon[76349]: from='client.77798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.850 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:43 smithi196 ceph-mon[76349]: from='client.77804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:43.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:43.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:43 smithi196 ceph-mon[76349]: from='client.77810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:44.006 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:42:44.006 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:42:44.006 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:42:44.006 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:42:44.006 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:42:44.007 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:42:44.322 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:42:44.322 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:44.322 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:44.322 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:44.322 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:44.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2082513442' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:44.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:44 smithi196 ceph-mon[76349]: from='client.77822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:44.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:44.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3695623623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:44.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:44 smithi196 ceph-mon[76349]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2082513442' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[100622]: from='client.77822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3695623623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[100622]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2082513442' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[102844]: from='client.77822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3695623623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:44 smithi184 ceph-mon[102844]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:46 smithi196 ceph-mon[76349]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:46 smithi184 ceph-mon[100622]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:46 smithi184 ceph-mon[102844]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:48 smithi196 ceph-mon[76349]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:48.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:48 smithi184 ceph-mon[100622]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:48.911 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:48 smithi184 ceph-mon[102844]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:50 smithi196 ceph-mon[76349]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:51.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:50 smithi184 ceph-mon[100622]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:51.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:50 smithi184 ceph-mon[102844]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:51.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:51.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:51.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:51.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:42:51.936 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:42:51.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:42:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:42:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:42:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:52 smithi196 ceph-mon[76349]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:52 smithi184 ceph-mon[100622]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:52 smithi184 ceph-mon[102844]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:54 smithi184 ceph-mon[100622]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:54 smithi184 ceph-mon[102844]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:55.105 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:54 smithi196 ceph-mon[76349]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[100622]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[102844]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:57 smithi196 ceph-mon[76349]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:42:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:00 smithi196 ceph-mon[76349]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:00 smithi184 ceph-mon[100622]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:00 smithi184 ceph-mon[102844]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:01.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:01.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:01.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:01.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:43:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:02 smithi196 ceph-mon[76349]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:02.766 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:02 smithi184 ceph-mon[100622]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:02.766 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:02 smithi184 ceph-mon[102844]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:04.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:04 smithi196 ceph-mon[76349]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:04.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:04 smithi184 ceph-mon[100622]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:04 smithi184 ceph-mon[102844]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:06 smithi196 ceph-mon[76349]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:06 smithi184 ceph-mon[100622]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:06 smithi184 ceph-mon[102844]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:08.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:08 smithi184 ceph-mon[100622]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:08.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:08 smithi184 ceph-mon[102844]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:08.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:08 smithi196 ceph-mon[76349]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:10 smithi184 ceph-mon[100622]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:10.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:10 smithi184 ceph-mon[102844]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:10 smithi196 ceph-mon[76349]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:11.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:11.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:11.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:43:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:12 smithi196 ceph-mon[76349]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:12 smithi184 ceph-mon[102844]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:12 smithi184 ceph-mon[100622]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:14.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[100622]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:14.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:14.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:14.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[102844]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:14.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:14.621 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:14.621 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:43:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:14 smithi196 ceph-mon[76349]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 5h - - 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 5h - - 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 18s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 16s ago 5h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 18s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 16s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 18s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:43:15.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 18s ago 6h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 16s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 18s ago 5h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 18s ago 5h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 16s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 18s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 18s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 18s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 16s ago 5h 72.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:43:15.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:43:15.144 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:43:15.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:43:15.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:43:15.725 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:43:15.726 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:43:15.726 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:43:15.726 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:43:16.034 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:43:16.035 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:16.035 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:16.035 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:16.035 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: from='client.67869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: from='client.77840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: from='client.77846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/700610897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1536940683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: from='client.67869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: from='client.77840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: from='client.77846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/700610897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1536940683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: from='client.67869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: from='client.77840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: from='client.77846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/700610897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1536940683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:17.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:17 smithi196 ceph-mon[76349]: from='client.77858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[100622]: from='client.77858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[102844]: from='client.77858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:18 smithi196 ceph-mon[76349]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:18 smithi184 ceph-mon[100622]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:18 smithi184 ceph-mon[102844]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:20 smithi184 ceph-mon[100622]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:20 smithi184 ceph-mon[102844]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:20 smithi196 ceph-mon[76349]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:21.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:21.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:21.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:43:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:22 smithi184 ceph-mon[100622]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:22 smithi184 ceph-mon[102844]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:22 smithi196 ceph-mon[76349]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:24 smithi196 ceph-mon[76349]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:24 smithi184 ceph-mon[100622]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:24 smithi184 ceph-mon[102844]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:26 smithi184 ceph-mon[100622]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:26 smithi184 ceph-mon[102844]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:26 smithi196 ceph-mon[76349]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:28 smithi184 ceph-mon[100622]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:28 smithi184 ceph-mon[102844]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:28 smithi196 ceph-mon[76349]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:30.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:30 smithi184 ceph-mon[100622]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:30.732 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:30 smithi184 ceph-mon[102844]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:30 smithi196 ceph-mon[76349]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:31.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:31.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:31.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:43:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:32 smithi184 ceph-mon[100622]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:32 smithi184 ceph-mon[102844]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:32 smithi196 ceph-mon[76349]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:34 smithi184 ceph-mon[100622]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:34 smithi184 ceph-mon[102844]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:34 smithi196 ceph-mon[76349]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:36 smithi184 ceph-mon[100622]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:36 smithi184 ceph-mon[102844]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:36 smithi196 ceph-mon[76349]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[100622]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[102844]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:37 smithi196 ceph-mon[76349]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:40 smithi196 ceph-mon[76349]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:40 smithi184 ceph-mon[100622]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:40 smithi184 ceph-mon[102844]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:41.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:41.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:41.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:43:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:42 smithi196 ceph-mon[76349]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:42 smithi184 ceph-mon[100622]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:42 smithi184 ceph-mon[102844]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:44.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:44 smithi196 ceph-mon[76349]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:44.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:44.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[100622]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[102844]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:46.329 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:43:46.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:46 smithi196 ceph-mon[76349]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:46.858 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:46 smithi184 ceph-mon[100622]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:46.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:46 smithi184 ceph-mon[102844]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 9s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 9s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 5h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 9s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:43:46.859 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 9s ago 6h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 9s ago 5h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 9s ago 5h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 67.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:43:46.860 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:43:46.861 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:43:47.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:43:47.172 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:43:47.438 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:43:47.439 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:43:47.747 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:43:47.747 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:47.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:47.748 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:47.748 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:47.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: from='client.77870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.748 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: from='client.77876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: from='client.77882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4093546417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[102844]: from='client.77894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: from='client.77870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.749 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: from='client.77876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:47.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: from='client.77882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4093546417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:47.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:47 smithi184 ceph-mon[100622]: from='client.77894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: from='client.77870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: from='client.77876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: from='client.77882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4093546417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:47 smithi196 ceph-mon[76349]: from='client.77894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3274935065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3274935065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3274935065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:49.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:49 smithi196 ceph-mon[76349]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:49.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:49 smithi184 ceph-mon[100622]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:49.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:49 smithi184 ceph-mon[102844]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:51 smithi196 ceph-mon[76349]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:52.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:51 smithi184 ceph-mon[100622]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:52.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:51 smithi184 ceph-mon[102844]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:51.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:51.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:43:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:43:51.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:43:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:43:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:43:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:53 smithi184 ceph-mon[100622]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:53.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:53 smithi184 ceph-mon[102844]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:53.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:53 smithi196 ceph-mon[76349]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:53.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:55 smithi196 ceph-mon[76349]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:55 smithi184 ceph-mon[100622]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:55 smithi184 ceph-mon[102844]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:58 smithi196 ceph-mon[76349]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[100622]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[102844]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:43:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[100622]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[102844]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:59 smithi196 ceph-mon[76349]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:01 smithi184 ceph-mon[100622]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:01 smithi184 ceph-mon[102844]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:01.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:01.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:01.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:02.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:01 smithi196 ceph-mon[76349]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:44:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:03 smithi184 ceph-mon[100622]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:03 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:03 smithi184 ceph-mon[102844]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:03 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:04.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:03 smithi196 ceph-mon[76349]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:04.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:03 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:06.004 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:05 smithi196 ceph-mon[76349]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:05 smithi184 ceph-mon[100622]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:05 smithi184 ceph-mon[102844]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:07.854 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:07 smithi196 ceph-mon[76349]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:44:08.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:07 smithi184 ceph-mon[100622]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:44:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:07 smithi184 ceph-mon[102844]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:44:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:09 smithi184 ceph-mon[100622]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:44:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:09 smithi184 ceph-mon[102844]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:44:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:09 smithi196 ceph-mon[76349]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:44:11.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:11 smithi184 ceph-mon[100622]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:11.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:11 smithi184 ceph-mon[102844]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:11.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:11.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:11.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:11.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:11.958 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:11.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:11 smithi196 ceph-mon[76349]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:12.220 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:44:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:13 smithi184 ceph-mon[100622]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:13 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:13 smithi184 ceph-mon[102844]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:13 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:13 smithi196 ceph-mon[76349]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:44:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:13 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:15.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:15 smithi184 ceph-mon[100622]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:44:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:15 smithi184 ceph-mon[102844]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:44:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:15 smithi196 ceph-mon[76349]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:44:18.038 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[102844]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[100622]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 6h - - 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 1s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 19s ago 5h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:44:18.576 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 1s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 19s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 1s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 1s ago 6h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 19s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 1s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 1s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:44:18.577 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 19s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 1s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 5h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 5h 80.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 1s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 5h 89.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:44:18.578 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 19s ago 5h 68.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:44:18.579 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:44:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:18 smithi196 ceph-mon[76349]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:44:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:44:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:44:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:44:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:44:18.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:44:18.893 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:44:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:44:19.159 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:44:19.159 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:44:19.159 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:44:19.159 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:44:19.468 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:44:19.469 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:19.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:19.469 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:19.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='client.77906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='client.77912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='client.67953 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.469 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2941167322' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='client.77906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='client.77912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='client.67953 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.470 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2941167322' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='client.77906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='client.77912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='client.67953 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2941167322' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[102844]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[102844]: from='client.77930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1915231187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[100622]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[100622]: from='client.77930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1915231187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:20 smithi196 ceph-mon[76349]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:20 smithi196 ceph-mon[76349]: from='client.77930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1915231187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:21.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:21.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:21.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:22.282 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:22 smithi184 ceph-mon[100622]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:22.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:22 smithi184 ceph-mon[102844]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:22.282 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:44:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:22 smithi196 ceph-mon[76349]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:24 smithi184 ceph-mon[102844]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:24 smithi184 ceph-mon[100622]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:24 smithi196 ceph-mon[76349]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:44:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:26 smithi184 ceph-mon[102844]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:44:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:26 smithi184 ceph-mon[100622]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:44:26.599 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:26 smithi196 ceph-mon[76349]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:44:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:28 smithi184 ceph-mon[100622]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T01:44:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:28 smithi184 ceph-mon[102844]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T01:44:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:28 smithi196 ceph-mon[76349]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T01:44:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:30.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:30 smithi184 ceph-mon[100622]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:44:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:30 smithi184 ceph-mon[102844]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:44:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:30 smithi196 ceph-mon[76349]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:44:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:31.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:31.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:31.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:32 smithi184 ceph-mon[100622]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:44:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:32 smithi184 ceph-mon[102844]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:44:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:44:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:32 smithi196 ceph-mon[76349]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T01:44:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:34 smithi184 ceph-mon[100622]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:44:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:34 smithi184 ceph-mon[102844]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:44:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:34 smithi196 ceph-mon[76349]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:44:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:36 smithi184 ceph-mon[100622]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:44:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:36 smithi184 ceph-mon[102844]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:44:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:36 smithi196 ceph-mon[76349]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:44:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:38 smithi196 ceph-mon[76349]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:44:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[100622]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[102844]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[100622]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:44:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[102844]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:44:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:39 smithi196 ceph-mon[76349]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:44:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:41.966 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:41.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:41.966 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:41.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:41.966 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:41.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:41 smithi184 ceph-mon[100622]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T01:44:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:41 smithi184 ceph-mon[102844]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T01:44:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:44:42.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:41 smithi196 ceph-mon[76349]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T01:44:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:43 smithi184 ceph-mon[100622]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:44:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:43 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:43 smithi184 ceph-mon[102844]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:44:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:43 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:43 smithi196 ceph-mon[76349]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T01:44:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:43 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:45.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:45 smithi196 ceph-mon[76349]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:45 smithi184 ceph-mon[100622]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:46.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:45 smithi184 ceph-mon[102844]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 61 op/s 2024-09-21T01:44:48.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:47 smithi196 ceph-mon[76349]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:44:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:47 smithi184 ceph-mon[100622]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:44:48.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:47 smithi184 ceph-mon[102844]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:44:49.763 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:44:50.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:49 smithi184 ceph-mon[100622]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:44:50.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:49 smithi184 ceph-mon[102844]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 12s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 11s ago 5h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:44:50.289 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 12s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 6h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 11s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 12s ago 5h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 11s ago 5h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 5h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 11s ago 5h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 11s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 5h 95.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 5h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:44:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 11s ago 5h 68.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:44:50.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:44:50.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:44:50.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:44:50.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:44:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:49 smithi196 ceph-mon[76349]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:44:50.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:44:50.599 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:44:50.870 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:44:50.871 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:44:51.181 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:44:51.181 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:51.181 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:51.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[100622]: from='client.67977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[100622]: from='client.77948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[100622]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2324484309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[102844]: from='client.67977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[102844]: from='client.77948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[102844]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2324484309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:51.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:50 smithi196 ceph-mon[76349]: from='client.67977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:50 smithi196 ceph-mon[76349]: from='client.77948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:50 smithi196 ceph-mon[76349]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2324484309' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:51.945 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:51.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:51.945 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:51.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:44:51.945 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:44:51.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:44:52.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[100622]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:44:52.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[100622]: from='client.77966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:52.230 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1260285312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:52.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:44:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T01:44:52.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[102844]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:44:52.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[102844]: from='client.77966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:52.231 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1260285312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:51 smithi196 ceph-mon[76349]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-21T01:44:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:51 smithi196 ceph-mon[76349]: from='client.77966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1260285312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:53 smithi184 ceph-mon[100622]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:53 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:54.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:53 smithi184 ceph-mon[102844]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:53 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:53 smithi196 ceph-mon[76349]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:53 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:55 smithi184 ceph-mon[100622]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:55 smithi184 ceph-mon[102844]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:55 smithi196 ceph-mon[76349]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[100622]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[102844]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:44:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:58 smithi196 ceph-mon[76349]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:44:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:44:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:00 smithi196 ceph-mon[76349]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[100622]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[102844]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:01.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:01.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:01.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:02.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:02 smithi184 ceph-mon[100622]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:02.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:02 smithi184 ceph-mon[102844]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:45:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:02 smithi196 ceph-mon[76349]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:04 smithi196 ceph-mon[76349]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:04 smithi184 ceph-mon[100622]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:04 smithi184 ceph-mon[102844]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:06 smithi196 ceph-mon[76349]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:06 smithi184 ceph-mon[100622]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:06.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:06 smithi184 ceph-mon[102844]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:08.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:08 smithi196 ceph-mon[76349]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:08 smithi184 ceph-mon[100622]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:08 smithi184 ceph-mon[102844]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:10 smithi196 ceph-mon[76349]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:10.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:10 smithi184 ceph-mon[102844]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:10 smithi184 ceph-mon[100622]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:11.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:11.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:11.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:12.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:12 smithi184 ceph-mon[102844]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:45:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:12 smithi184 ceph-mon[100622]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:12 smithi196 ceph-mon[76349]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[100622]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[102844]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:14.238 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:14 smithi196 ceph-mon[76349]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:16 smithi196 ceph-mon[76349]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:16 smithi184 ceph-mon[100622]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:16 smithi184 ceph-mon[102844]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:18 smithi196 ceph-mon[76349]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[100622]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[102844]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[100622]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[102844]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:20 smithi196 ceph-mon[76349]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:45:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:21.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:21.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:21.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 6h - - 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 4s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 2s ago 5h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 4s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:45:22.004 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 4s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 4s ago 6h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 2s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 4s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 4s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 2s ago 5h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 4s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 4s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:45:22.005 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:45:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 2s ago 5h 69.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:45:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:45:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:45:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:45:22.006 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:45:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:45:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:22 smithi184 ceph-mon[100622]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:22 smithi184 ceph-mon[100622]: from='client.68013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:22 smithi184 ceph-mon[102844]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:22 smithi184 ceph-mon[102844]: from='client.68013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:45:22.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:45:22.317 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:45:22.584 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:45:22.584 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:45:22.584 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:45:22.584 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:45:22.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:45:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:22 smithi196 ceph-mon[76349]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:22 smithi196 ceph-mon[76349]: from='client.68013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:22.893 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:45:22.893 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:22.893 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:22.893 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:22.893 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[100622]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[100622]: from='client.77990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2276636238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[100622]: from='client.78002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2171669137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:23.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[102844]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[102844]: from='client.77990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2276636238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:23.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[102844]: from='client.78002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2171669137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:23 smithi196 ceph-mon[76349]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:23 smithi196 ceph-mon[76349]: from='client.77990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2276636238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:23 smithi196 ceph-mon[76349]: from='client.78002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2171669137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:24 smithi184 ceph-mon[102844]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:24 smithi184 ceph-mon[100622]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:24 smithi196 ceph-mon[76349]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:26 smithi196 ceph-mon[76349]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:26 smithi184 ceph-mon[100622]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:26 smithi184 ceph-mon[102844]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:28.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:28 smithi196 ceph-mon[76349]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:28 smithi184 ceph-mon[100622]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:28 smithi184 ceph-mon[102844]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:29.411 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:30 smithi196 ceph-mon[76349]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:30 smithi184 ceph-mon[102844]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:30 smithi184 ceph-mon[100622]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:31.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:31.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:31.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:45:32.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:32 smithi184 ceph-mon[100622]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:32.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:32 smithi184 ceph-mon[102844]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:32 smithi196 ceph-mon[76349]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:34.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:34 smithi184 ceph-mon[100622]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:34.622 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:34.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:34 smithi184 ceph-mon[102844]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:34.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:34 smithi196 ceph-mon[76349]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:36 smithi184 ceph-mon[100622]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:36 smithi184 ceph-mon[102844]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:36 smithi196 ceph-mon[76349]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[102844]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[100622]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:38 smithi196 ceph-mon[76349]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:40 smithi196 ceph-mon[76349]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[100622]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[102844]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:41.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:41.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:41.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:45:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:42 smithi196 ceph-mon[76349]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:42 smithi184 ceph-mon[100622]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:42 smithi184 ceph-mon[102844]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:44 smithi196 ceph-mon[76349]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[100622]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[102844]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:46 smithi196 ceph-mon[76349]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:46 smithi184 ceph-mon[100622]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:46 smithi184 ceph-mon[102844]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:48 smithi196 ceph-mon[76349]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:48 smithi184 ceph-mon[100622]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:48 smithi184 ceph-mon[102844]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:50.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:50 smithi196 ceph-mon[76349]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:50 smithi184 ceph-mon[100622]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:50 smithi184 ceph-mon[102844]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:51.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:51.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:45:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:45:51.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:45:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:45:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:45:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:52 smithi196 ceph-mon[76349]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:52 smithi184 ceph-mon[100622]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:52 smithi184 ceph-mon[102844]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:53.187 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 15s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 14s ago 5h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 15s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 15s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 6h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:45:53.714 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 14s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 15s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 14s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 14s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 14s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 14s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 14s ago 5h 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:45:53.715 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:45:53.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:45:53.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:45:54.032 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:45:54.310 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:45:54.311 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:45:54.311 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:45:54.311 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:45:54.311 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:45:54.626 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:45:54.626 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:54.626 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[100622]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[100622]: from='client.68049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[100622]: from='client.68055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3429131249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[102844]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:54.627 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[102844]: from='client.68049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[102844]: from='client.68055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:54.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3429131249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:54 smithi196 ceph-mon[76349]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:54 smithi196 ceph-mon[76349]: from='client.68049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:54 smithi196 ceph-mon[76349]: from='client.68055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3429131249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:55.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[100622]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.616 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[100622]: from='client.68073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.617 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/498939049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:55.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[102844]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[102844]: from='client.68073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.617 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/498939049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:55 smithi196 ceph-mon[76349]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:55 smithi196 ceph-mon[76349]: from='client.68073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/498939049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:56.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:56 smithi184 ceph-mon[100622]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:56.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:56 smithi184 ceph-mon[102844]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:56 smithi196 ceph-mon[76349]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[102844]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[100622]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:58 smithi196 ceph-mon[76349]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:45:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[100622]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[102844]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:00 smithi196 ceph-mon[76349]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:01.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:01.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:01.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:46:02.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:02 smithi184 ceph-mon[102844]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:02.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:02 smithi184 ceph-mon[100622]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:02 smithi196 ceph-mon[76349]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:04 smithi184 ceph-mon[100622]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:04 smithi184 ceph-mon[102844]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:04 smithi196 ceph-mon[76349]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:06 smithi184 ceph-mon[100622]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:06 smithi184 ceph-mon[102844]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:06 smithi196 ceph-mon[76349]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:08 smithi184 ceph-mon[100622]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:08 smithi184 ceph-mon[102844]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:08.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:08 smithi196 ceph-mon[76349]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:10 smithi196 ceph-mon[76349]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:10 smithi184 ceph-mon[100622]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:10 smithi184 ceph-mon[102844]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:11.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:11.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:11.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:46:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:12 smithi184 ceph-mon[100622]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:12 smithi184 ceph-mon[102844]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:12 smithi196 ceph-mon[76349]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:14.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[100622]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:14.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:14.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:14.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[102844]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:14.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:14.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:14 smithi196 ceph-mon[76349]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:16 smithi184 ceph-mon[100622]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:16 smithi184 ceph-mon[102844]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:16 smithi196 ceph-mon[76349]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[100622]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[102844]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:18 smithi196 ceph-mon[76349]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[100622]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[102844]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:20 smithi196 ceph-mon[76349]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:21.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:21.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:21.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:46:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:22 smithi184 ceph-mon[100622]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:22 smithi184 ceph-mon[102844]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:22 smithi196 ceph-mon[76349]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:24 smithi184 ceph-mon[100622]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:24 smithi184 ceph-mon[102844]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:24 smithi196 ceph-mon[76349]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:24.924 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:46:25.453 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:46:25.453 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T01:46:25.453 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T01:46:25.453 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 7s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:46:25.453 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 5s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 7s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 7s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 7s ago 6h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 7s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 7s ago 5h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 5s ago 5h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 7s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 7s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 7s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 7s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:46:25.454 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 5s ago 5h 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:46:25.455 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:25.767 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:46:25.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:46:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:46:26.037 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:46:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:46:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: from='client.78050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: from='client.78056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: from='client.78062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1094269000' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2349955228' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: from='client.78050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: from='client.78056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: from='client.78062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1094269000' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2349955228' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: from='client.78050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: from='client.78056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: from='client.78062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1094269000' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2349955228' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:27 smithi184 ceph-mon[100622]: from='client.78074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:27 smithi184 ceph-mon[102844]: from='client.78074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:27.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:27 smithi196 ceph-mon[76349]: from='client.78074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:28 smithi184 ceph-mon[100622]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:28 smithi184 ceph-mon[102844]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:28 smithi196 ceph-mon[76349]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:29.744 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:30 smithi196 ceph-mon[76349]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:30 smithi184 ceph-mon[100622]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:30 smithi184 ceph-mon[102844]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:31.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:31.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:31.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:46:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:32 smithi184 ceph-mon[100622]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:32 smithi184 ceph-mon[102844]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:32 smithi196 ceph-mon[76349]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:34 smithi184 ceph-mon[100622]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:34 smithi184 ceph-mon[102844]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:34 smithi196 ceph-mon[76349]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:36 smithi184 ceph-mon[100622]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:36 smithi184 ceph-mon[102844]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:36 smithi196 ceph-mon[76349]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:38 smithi196 ceph-mon[76349]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[100622]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[102844]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:40 smithi196 ceph-mon[76349]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[102844]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[100622]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:41.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:41.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:41.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:46:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:42 smithi196 ceph-mon[76349]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:42 smithi184 ceph-mon[102844]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:42 smithi184 ceph-mon[100622]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:44 smithi196 ceph-mon[76349]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[100622]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[102844]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:46 smithi196 ceph-mon[76349]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:46 smithi184 ceph-mon[102844]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:46 smithi184 ceph-mon[100622]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:48 smithi196 ceph-mon[76349]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:48 smithi184 ceph-mon[100622]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:48 smithi184 ceph-mon[102844]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:50 smithi196 ceph-mon[76349]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:50 smithi184 ceph-mon[100622]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:50 smithi184 ceph-mon[102844]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:51.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:51.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:46:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:46:51.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:46:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:46:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:46:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:52 smithi196 ceph-mon[76349]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:52 smithi184 ceph-mon[102844]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:52 smithi184 ceph-mon[100622]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:54 smithi196 ceph-mon[76349]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:54 smithi184 ceph-mon[102844]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:54 smithi184 ceph-mon[100622]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:56.658 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:46:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:56 smithi196 ceph-mon[76349]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:56.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:56 smithi184 ceph-mon[102844]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:56.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:56 smithi184 ceph-mon[100622]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 6h - - 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 19s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 17s ago 5h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 6h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:46:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 17s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 19s ago 5h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 17s ago 5h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 17s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 17s ago 5h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 17s ago 5h 72.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:46:57.203 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 17s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:46:57.522 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:46:57.522 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:46:57.523 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:46:57.524 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:46:57.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1802365393' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:57.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1802365393' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:46:57.811 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:46:57.812 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:46:57.812 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:46:57.812 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:46:57.812 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:46:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1802365393' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:58.136 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:46:58.136 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:58.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:58.136 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:58.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:58.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='client.68121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.782 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='client.68133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/647977063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='client.68121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='client.68133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/647977063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:58.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='client.68121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='client.68133 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/647977063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:46:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:59 smithi196 ceph-mon[76349]: from='client.78110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:59 smithi184 ceph-mon[100622]: from='client.78110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:59 smithi184 ceph-mon[102844]: from='client.78110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:00 smithi196 ceph-mon[76349]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[102844]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[100622]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:01.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:01.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:01.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:47:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:02 smithi196 ceph-mon[76349]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:02 smithi184 ceph-mon[100622]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:02 smithi184 ceph-mon[102844]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:04 smithi196 ceph-mon[76349]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:05.052 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:04 smithi184 ceph-mon[100622]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:05.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:04 smithi184 ceph-mon[102844]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:06 smithi196 ceph-mon[76349]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:06 smithi184 ceph-mon[100622]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:06 smithi184 ceph-mon[102844]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:08 smithi196 ceph-mon[76349]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:08 smithi184 ceph-mon[100622]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:08 smithi184 ceph-mon[102844]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:09.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:09 smithi196 ceph-mon[76349]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:09 smithi184 ceph-mon[100622]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:09 smithi184 ceph-mon[102844]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:11 smithi184 ceph-mon[100622]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:11 smithi184 ceph-mon[102844]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:11.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:11.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:11.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:12.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:11 smithi196 ceph-mon[76349]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:47:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:13 smithi184 ceph-mon[100622]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:13 smithi184 ceph-mon[102844]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:13 smithi196 ceph-mon[76349]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:15 smithi196 ceph-mon[76349]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:15 smithi184 ceph-mon[100622]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:15 smithi184 ceph-mon[102844]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:17.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:17 smithi184 ceph-mon[100622]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:17.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:17 smithi184 ceph-mon[102844]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:17 smithi196 ceph-mon[76349]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:19.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:19.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:20 smithi196 ceph-mon[76349]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[100622]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[102844]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:21.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:21.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:21.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:47:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:22 smithi196 ceph-mon[76349]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:22 smithi184 ceph-mon[100622]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:22 smithi184 ceph-mon[102844]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:24 smithi196 ceph-mon[76349]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:24 smithi184 ceph-mon[102844]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:24 smithi184 ceph-mon[100622]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:26 smithi196 ceph-mon[76349]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:26 smithi184 ceph-mon[100622]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:26 smithi184 ceph-mon[102844]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:28.438 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:47:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:28 smithi196 ceph-mon[76349]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:28 smithi184 ceph-mon[100622]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:28 smithi184 ceph-mon[102844]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 10s ago 5h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 8s ago 5h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 10s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:47:28.965 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 10s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 10s ago 6h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 8s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 10s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 10s ago 5h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 8s ago 5h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 10s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 10s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 10s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 10s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:47:28.966 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 8s ago 5h 72.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:47:28.967 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:47:29.278 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:47:29.278 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:47:29.278 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:47:29.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:47:29.279 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[100622]: from='client.68157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/372798894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[102844]: from='client.68157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/372798894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:29.555 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:47:29.556 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:47:29.866 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:47:29.866 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:29.866 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:29.866 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:29.866 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:29 smithi196 ceph-mon[76349]: from='client.68157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/372798894' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[100622]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[100622]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[100622]: from='client.78134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[100622]: from='client.68181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1456306187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[102844]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[102844]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[102844]: from='client.78134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[102844]: from='client.68181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1456306187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:30 smithi196 ceph-mon[76349]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:30 smithi196 ceph-mon[76349]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:30 smithi196 ceph-mon[76349]: from='client.78134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:30 smithi196 ceph-mon[76349]: from='client.68181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1456306187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:31.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:31.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:31.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:47:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:32 smithi196 ceph-mon[76349]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:32 smithi184 ceph-mon[100622]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:32 smithi184 ceph-mon[102844]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:34 smithi184 ceph-mon[100622]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:34 smithi184 ceph-mon[102844]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:34 smithi196 ceph-mon[76349]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:36 smithi184 ceph-mon[100622]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:36 smithi184 ceph-mon[102844]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:36 smithi196 ceph-mon[76349]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:38.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:38 smithi184 ceph-mon[102844]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:38 smithi184 ceph-mon[100622]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:38 smithi196 ceph-mon[76349]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[100622]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[102844]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:40 smithi196 ceph-mon[76349]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:41.701Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:41.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:41.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:47:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:42 smithi184 ceph-mon[100622]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:42 smithi184 ceph-mon[102844]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:42 smithi196 ceph-mon[76349]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:44 smithi184 ceph-mon[100622]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:44 smithi184 ceph-mon[102844]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:44 smithi196 ceph-mon[76349]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:46.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:46 smithi196 ceph-mon[76349]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:46 smithi184 ceph-mon[100622]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:46 smithi184 ceph-mon[102844]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:48 smithi184 ceph-mon[100622]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:48 smithi184 ceph-mon[102844]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:48 smithi196 ceph-mon[76349]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:50 smithi184 ceph-mon[100622]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:50.817 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:50 smithi184 ceph-mon[102844]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:50 smithi196 ceph-mon[76349]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:51.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:51.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:47:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:47:51.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:47:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:47:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:47:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:52 smithi196 ceph-mon[76349]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:52 smithi184 ceph-mon[100622]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:52 smithi184 ceph-mon[102844]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:54 smithi184 ceph-mon[100622]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:54 smithi184 ceph-mon[102844]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:54 smithi196 ceph-mon[76349]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:56 smithi184 ceph-mon[100622]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:56 smithi184 ceph-mon[102844]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:56 smithi196 ceph-mon[76349]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:58.663 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:58 smithi184 ceph-mon[100622]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:58.663 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:58 smithi184 ceph-mon[102844]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:58 smithi196 ceph-mon[76349]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:59.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.793 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:59.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.794 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:47:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:47:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:47:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:00.164 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:48:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:48:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T01:48:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 6h - - 2024-09-21T01:48:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 2s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:48:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (5h) 0s ago 5h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 0s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 6h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 0s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (5h) 2s ago 5h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (5h) 0s ago 5h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:48:00.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 0s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 0s ago 6h 72.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:48:00.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:48:00.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:48:00.734 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 0s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:48:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:00 smithi196 ceph-mon[76349]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[100622]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[102844]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:00.910 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:48:01.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:48:01.049 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:48:01.317 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:48:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:48:01.318 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:48:01.629 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:48:01.629 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:01.629 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:01.629 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:01.629 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:01 smithi196 ceph-mon[76349]: from='client.68193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:01 smithi196 ceph-mon[76349]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1864783637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:02.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[100622]: from='client.68193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[100622]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1864783637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[102844]: from='client.68193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[102844]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1864783637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:01.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:01.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:01.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[100622]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[100622]: from='client.78170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[100622]: from='client.68217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1226502509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[102844]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[102844]: from='client.78170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[102844]: from='client.68217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1226502509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:02 smithi196 ceph-mon[76349]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:02 smithi196 ceph-mon[76349]: from='client.78170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:02 smithi196 ceph-mon[76349]: from='client.68217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1226502509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:03 smithi196 ceph-mon[76349]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:03 smithi184 ceph-mon[100622]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:03 smithi184 ceph-mon[102844]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:05 smithi184 ceph-mon[100622]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:05 smithi184 ceph-mon[102844]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:05 smithi196 ceph-mon[76349]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:08.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:07 smithi184 ceph-mon[100622]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:07 smithi184 ceph-mon[102844]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:07 smithi196 ceph-mon[76349]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:09 smithi184 ceph-mon[100622]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:09 smithi184 ceph-mon[102844]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:10.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:09 smithi196 ceph-mon[76349]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:11 smithi184 ceph-mon[100622]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:12.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:11 smithi184 ceph-mon[102844]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:11.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:11.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:11.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:11 smithi196 ceph-mon[76349]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:48:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:13 smithi196 ceph-mon[76349]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:13.889 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:48:13.813078568Z level=info msg="Completed cleanup jobs" duration=13.538426ms 2024-09-21T01:48:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:13 smithi184 ceph-mon[100622]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:13 smithi184 ceph-mon[102844]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:15.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:15.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:15 smithi184 ceph-mon[100622]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:15 smithi184 ceph-mon[102844]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:15 smithi196 ceph-mon[76349]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:17 smithi184 ceph-mon[100622]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:18.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:17 smithi184 ceph-mon[102844]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:17 smithi196 ceph-mon[76349]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[100622]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[102844]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:19 smithi196 ceph-mon[76349]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:21.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:21.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:21.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:21.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:48:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:22 smithi184 ceph-mon[100622]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:22 smithi184 ceph-mon[102844]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:22 smithi196 ceph-mon[76349]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:24 smithi184 ceph-mon[100622]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:24 smithi184 ceph-mon[102844]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:24 smithi196 ceph-mon[76349]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:26 smithi196 ceph-mon[76349]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:26 smithi184 ceph-mon[100622]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:26 smithi184 ceph-mon[102844]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:28 smithi184 ceph-mon[102844]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:28 smithi184 ceph-mon[100622]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:28 smithi196 ceph-mon[76349]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:30 smithi184 ceph-mon[100622]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:30 smithi184 ceph-mon[102844]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:30 smithi196 ceph-mon[76349]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:31.931 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:48:32.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:31.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:32.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:31.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:32.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:31.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 6h - - 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 13s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 13s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:48:32.463 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 13s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 13s ago 6h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 13s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 13s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 13s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 13s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 13s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:48:32.464 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 6h 72.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:48:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:48:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:48:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:48:32.465 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 12s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:48:32.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:48:32.777 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:48:32.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:32 smithi184 ceph-mon[100622]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:32.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:32 smithi184 ceph-mon[102844]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:32 smithi196 ceph-mon[76349]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:33.044 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:48:33.044 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:48:33.045 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:48:33.358 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:48:33.358 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:33.358 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:33.358 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:33.358 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:33.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[100622]: from='client.68229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[100622]: from='client.68235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[100622]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2207573229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4288414366' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[102844]: from='client.68229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[102844]: from='client.68235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[102844]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2207573229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:33.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4288414366' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:33.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:33 smithi196 ceph-mon[76349]: from='client.68229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:33 smithi196 ceph-mon[76349]: from='client.68235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:33 smithi196 ceph-mon[76349]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:33.835 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2207573229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:33.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4288414366' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:34.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:34 smithi196 ceph-mon[76349]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:34.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:34 smithi196 ceph-mon[76349]: from='client.68253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:34 smithi184 ceph-mon[100622]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:34 smithi184 ceph-mon[100622]: from='client.68253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:34 smithi184 ceph-mon[102844]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:34 smithi184 ceph-mon[102844]: from='client.68253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:35.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:36 smithi184 ceph-mon[102844]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:36 smithi184 ceph-mon[100622]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:36 smithi196 ceph-mon[76349]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:38 smithi184 ceph-mon[100622]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:38 smithi184 ceph-mon[102844]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:38 smithi196 ceph-mon[76349]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[100622]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[102844]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:39 smithi196 ceph-mon[76349]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:41.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:41.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:41.798 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:41.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:48:42.220 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:48:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:42 smithi184 ceph-mon[100622]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:42 smithi184 ceph-mon[102844]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:42 smithi196 ceph-mon[76349]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:44 smithi184 ceph-mon[102844]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:44 smithi184 ceph-mon[100622]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:44 smithi196 ceph-mon[76349]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:46.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:46 smithi196 ceph-mon[76349]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:46 smithi184 ceph-mon[100622]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:46 smithi184 ceph-mon[102844]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:48 smithi184 ceph-mon[100622]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:48 smithi184 ceph-mon[102844]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:48 smithi196 ceph-mon[76349]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:50 smithi184 ceph-mon[100622]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:50 smithi184 ceph-mon[102844]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:50 smithi196 ceph-mon[76349]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:51.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:51.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:48:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:48:51.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:48:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:48:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:48:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:52 smithi184 ceph-mon[100622]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:52 smithi184 ceph-mon[102844]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:52 smithi196 ceph-mon[76349]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:54.772 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:54 smithi196 ceph-mon[76349]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:54 smithi184 ceph-mon[100622]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:54 smithi184 ceph-mon[102844]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:55.871 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:56 smithi196 ceph-mon[76349]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:56 smithi184 ceph-mon[100622]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:56 smithi184 ceph-mon[102844]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:58 smithi184 ceph-mon[100622]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:58 smithi184 ceph-mon[102844]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:58 smithi196 ceph-mon[76349]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[100622]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[102844]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:48:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:59 smithi196 ceph-mon[76349]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:48:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:01.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[100622]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:01.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-mon[102844]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:01.897 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:01.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:01.898 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:01.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:01.898 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:01.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:02.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:01 smithi196 ceph-mon[76349]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:02.169 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:49:03.643 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:49:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:03 smithi184 ceph-mon[100622]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:04.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:03 smithi184 ceph-mon[100622]: from='client.78230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:03 smithi184 ceph-mon[102844]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:04.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:03 smithi184 ceph-mon[102844]: from='client.78230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:04.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:03 smithi196 ceph-mon[76349]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:03 smithi196 ceph-mon[76349]: from='client.78230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 6h - - 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (5h) 5s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 3s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:49:04.174 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 6h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 3s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 3s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 3s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 3s ago 6h 72.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:49:04.175 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:49:04.176 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 3s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:49:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:49:04.484 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:49:04.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:49:04.486 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:49:04.752 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:49:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:49:04.754 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[102844]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[102844]: from='client.78242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3449280613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[100622]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[100622]: from='client.78242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3449280613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:05.063 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:49:05.063 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:05.063 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:05.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:04 smithi196 ceph-mon[76349]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:04 smithi196 ceph-mon[76349]: from='client.78242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3449280613' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[100622]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[100622]: from='client.68289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3253239826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[102844]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[102844]: from='client.68289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3253239826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:05 smithi196 ceph-mon[76349]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:05 smithi196 ceph-mon[76349]: from='client.68289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3253239826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:08.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:07 smithi184 ceph-mon[100622]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:07 smithi184 ceph-mon[102844]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:08.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:07 smithi196 ceph-mon[76349]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:09 smithi184 ceph-mon[100622]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:09 smithi184 ceph-mon[102844]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:10.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:09 smithi196 ceph-mon[76349]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:11 smithi184 ceph-mon[100622]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:11 smithi184 ceph-mon[102844]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:11.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:11.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:11.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:12.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:11 smithi196 ceph-mon[76349]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:49:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:13 smithi184 ceph-mon[100622]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:13 smithi184 ceph-mon[102844]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:13 smithi196 ceph-mon[76349]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:15 smithi184 ceph-mon[100622]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:15 smithi184 ceph-mon[102844]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:15 smithi196 ceph-mon[76349]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:17 smithi184 ceph-mon[100622]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:18.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:17 smithi184 ceph-mon[102844]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:17 smithi196 ceph-mon[76349]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:20 smithi196 ceph-mon[76349]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[102844]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[100622]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.980 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[100622]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[102844]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:21.981 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:21.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:21.982 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:21.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:21.982 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:21.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:21 smithi196 ceph-mon[76349]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:22.232 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:49:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:23 smithi184 ceph-mon[100622]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:23 smithi184 ceph-mon[102844]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:24.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:23 smithi196 ceph-mon[76349]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:25 smithi184 ceph-mon[100622]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:25 smithi184 ceph-mon[102844]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:25 smithi196 ceph-mon[76349]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:28.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:27 smithi184 ceph-mon[100622]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:28.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:27 smithi184 ceph-mon[102844]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:27 smithi196 ceph-mon[76349]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:29 smithi184 ceph-mon[100622]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:29 smithi184 ceph-mon[102844]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:29 smithi196 ceph-mon[76349]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:32.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:31 smithi184 ceph-mon[100622]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:31 smithi184 ceph-mon[102844]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:31.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:31.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:31.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:31 smithi196 ceph-mon[76349]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:49:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:33 smithi184 ceph-mon[100622]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:34.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:33 smithi184 ceph-mon[102844]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:33 smithi196 ceph-mon[76349]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:49:35.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:35.361 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 6h - - 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 16s ago 5h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:49:35.894 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 16s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 16s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 16s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 16s ago 6h 92.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 16s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 16s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 16s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:49:35.895 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 6h 73.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:49:35.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:49:35.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:49:35.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:49:35.896 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 5h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:49:36.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:35 smithi196 ceph-mon[76349]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:36.007 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:35 smithi196 ceph-mon[76349]: from='client.68301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.008 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:35 smithi196 ceph-mon[76349]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:49:36.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:49:36.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[100622]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:36.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[100622]: from='client.68301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.203 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[100622]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[102844]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:36.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[102844]: from='client.68301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:35 smithi184 ceph-mon[102844]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:49:36.476 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:49:36.477 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:49:36.477 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:49:36.785 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:49:36.785 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:36.785 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:36.786 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:36.786 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[100622]: from='client.78278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4273219758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[100622]: from='client.78290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2579238851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[102844]: from='client.78278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4273219758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[102844]: from='client.78290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2579238851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:36 smithi196 ceph-mon[76349]: from='client.78278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4273219758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:36 smithi196 ceph-mon[76349]: from='client.78290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2579238851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:37 smithi196 ceph-mon[76349]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:37 smithi184 ceph-mon[100622]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:37 smithi184 ceph-mon[102844]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[102844]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[100622]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:40 smithi196 ceph-mon[76349]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:41.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:41.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:41.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:41.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:41.893 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:41.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:49:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[100622]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[102844]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:41 smithi196 ceph-mon[76349]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:49:44.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:43 smithi184 ceph-mon[100622]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:44.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:43 smithi184 ceph-mon[102844]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:43 smithi196 ceph-mon[76349]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:46.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:45 smithi196 ceph-mon[76349]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:45 smithi184 ceph-mon[100622]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:46.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:45 smithi184 ceph-mon[102844]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:47 smithi184 ceph-mon[100622]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:48.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:47 smithi184 ceph-mon[102844]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:47 smithi196 ceph-mon[76349]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:49 smithi184 ceph-mon[100622]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:49 smithi184 ceph-mon[102844]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:49 smithi196 ceph-mon[76349]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:52.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:51 smithi184 ceph-mon[100622]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:52.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:51 smithi184 ceph-mon[102844]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:51.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:51.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:49:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:49:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:49:51.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:49:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:49:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:51 smithi196 ceph-mon[76349]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:53 smithi184 ceph-mon[100622]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:54.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:53 smithi184 ceph-mon[102844]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:54.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:53 smithi196 ceph-mon[76349]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:55 smithi196 ceph-mon[76349]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:55 smithi184 ceph-mon[100622]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:55 smithi184 ceph-mon[102844]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:58.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:57 smithi196 ceph-mon[76349]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:57 smithi184 ceph-mon[100622]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:57 smithi184 ceph-mon[102844]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[102844]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[100622]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:49:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:49:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:59 smithi196 ceph-mon[76349]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:49:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:01.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:01.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:01.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[100622]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.258 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[102844]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:02.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.259 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:02 smithi196 ceph-mon[76349]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:04.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:04 smithi184 ceph-mon[100622]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:04 smithi184 ceph-mon[102844]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:04.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:04 smithi196 ceph-mon[76349]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:05.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:06 smithi184 ceph-mon[100622]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:06.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:06 smithi184 ceph-mon[102844]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:06.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:06 smithi196 ceph-mon[76349]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:07.082 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:50:07.605 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:50:07.605 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 6h - - 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 6s ago 6h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 6s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:50:07.606 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 6s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 6s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:50:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 6s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 6s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 6s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:50:07.608 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 6s ago 6h 74.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:50:07.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:50:07.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:50:07.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:50:07.609 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:50:07.920 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:50:07.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:50:07.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:50:07.922 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:50:08.191 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:50:08.192 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:50:08.192 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:50:08.192 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[102844]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[102844]: from='client.78302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[102844]: from='client.68343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[102844]: from='client.78314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4095168984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:08.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[100622]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:08.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[100622]: from='client.78302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[100622]: from='client.68343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[100622]: from='client.78314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4095168984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:08 smithi196 ceph-mon[76349]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:08 smithi196 ceph-mon[76349]: from='client.78302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:08 smithi196 ceph-mon[76349]: from='client.68343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:08 smithi196 ceph-mon[76349]: from='client.78314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4095168984' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:08.501 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:08.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:08.502 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:08.502 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:08.502 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:09 smithi184 ceph-mon[100622]: from='client.68361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2969754065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:09 smithi184 ceph-mon[102844]: from='client.68361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2969754065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:09 smithi196 ceph-mon[76349]: from='client.68361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2969754065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:10 smithi184 ceph-mon[100622]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:10.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:10 smithi184 ceph-mon[102844]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:10.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:10 smithi196 ceph-mon[76349]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:11.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:11.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:11.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:12.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:12 smithi184 ceph-mon[100622]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:12.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:12 smithi184 ceph-mon[102844]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:50:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:12 smithi196 ceph-mon[76349]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:14 smithi196 ceph-mon[76349]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:14 smithi184 ceph-mon[100622]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:14 smithi184 ceph-mon[102844]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:15.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:15.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:16.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:16 smithi196 ceph-mon[76349]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:16 smithi184 ceph-mon[100622]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:16 smithi184 ceph-mon[102844]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:18 smithi196 ceph-mon[76349]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:18 smithi184 ceph-mon[100622]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:18.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:18 smithi184 ceph-mon[102844]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[100622]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[102844]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:20 smithi196 ceph-mon[76349]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:21.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:21.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:21.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[100622]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:50:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[102844]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:22 smithi196 ceph-mon[76349]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:24.232 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:24 smithi184 ceph-mon[100622]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:24.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:24 smithi184 ceph-mon[102844]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:24 smithi196 ceph-mon[76349]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:26 smithi184 ceph-mon[100622]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:26 smithi184 ceph-mon[102844]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:26 smithi196 ceph-mon[76349]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:28 smithi184 ceph-mon[100622]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:28 smithi184 ceph-mon[102844]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:28 smithi196 ceph-mon[76349]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:30 smithi184 ceph-mon[100622]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:30 smithi184 ceph-mon[102844]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:30 smithi196 ceph-mon[76349]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:31.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:31.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:31.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:32] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-21T01:50:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:32 smithi184 ceph-mon[100622]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:32 smithi184 ceph-mon[102844]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:32 smithi196 ceph-mon[76349]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:34 smithi184 ceph-mon[100622]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:34 smithi184 ceph-mon[102844]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:34 smithi196 ceph-mon[76349]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:36.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:36 smithi196 ceph-mon[76349]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:36 smithi184 ceph-mon[100622]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:36 smithi184 ceph-mon[102844]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:38.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:38 smithi184 ceph-mon[102844]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:38.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:38 smithi184 ceph-mon[100622]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:38.798 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:50:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:38 smithi196 ceph-mon[76349]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 6h - - 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 19s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 19s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:50:39.323 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 19s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 19s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 19s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 19s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 19s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 19s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 19s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:50:39.324 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:50:39.325 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:50:39.325 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:50:39.325 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:50:39.662 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:50:39.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:50:39.664 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:50:39.664 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:50:39.664 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:50:39.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:50:39.950 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:50:39.951 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:50:40.261 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:40.261 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:40.261 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:40.261 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='client.68373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='client.78344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2925177297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2131622925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='client.68373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='client.78344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2925177297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2131622925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='client.68373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='client.78344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.051 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:41.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2925177297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:41.052 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2131622925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[100622]: from='client.68397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[102844]: from='client.68397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:41.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:41.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:41.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:41 smithi196 ceph-mon[76349]: from='client.68397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:50:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:42] "GET /metrics HTTP/1.1" 200 38938 "" "Prometheus/2.43.0" 2024-09-21T01:50:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:42 smithi184 ceph-mon[100622]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:42 smithi184 ceph-mon[102844]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:42 smithi196 ceph-mon[76349]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:44.886 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:44 smithi184 ceph-mon[100622]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:44.886 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:44.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:44 smithi184 ceph-mon[102844]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:44.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:44 smithi196 ceph-mon[76349]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:46.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:46 smithi184 ceph-mon[100622]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:46 smithi184 ceph-mon[102844]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:46 smithi196 ceph-mon[76349]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:48.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:47 smithi184 ceph-mon[100622]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:48.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:47 smithi184 ceph-mon[102844]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:47 smithi196 ceph-mon[76349]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:50.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:49 smithi184 ceph-mon[100622]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:50.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:49 smithi184 ceph-mon[102844]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:49 smithi196 ceph-mon[76349]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:52.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:51 smithi184 ceph-mon[100622]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:52.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:51 smithi184 ceph-mon[102844]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:51.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:51.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:50:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:50:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:50:51.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:50:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:51 smithi196 ceph-mon[76349]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:50:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:50:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:53 smithi196 ceph-mon[76349]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:53 smithi184 ceph-mon[100622]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:54.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:53 smithi184 ceph-mon[102844]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:55.327 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:56.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:55 smithi196 ceph-mon[76349]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:55 smithi184 ceph-mon[100622]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:55 smithi184 ceph-mon[102844]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:57.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:57 smithi196 ceph-mon[76349]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:57 smithi184 ceph-mon[100622]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:57 smithi184 ceph-mon[102844]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[100622]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[102844]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:50:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:59 smithi196 ceph-mon[76349]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:50:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:01.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:01.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:01.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:51:02.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[100622]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:02.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[102844]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:02.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:02 smithi196 ceph-mon[76349]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:04.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:04 smithi184 ceph-mon[100622]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:04.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:04 smithi184 ceph-mon[102844]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:04 smithi196 ceph-mon[76349]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:05.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:05.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:06.791 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:06 smithi184 ceph-mon[102844]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:06.791 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:06 smithi184 ceph-mon[100622]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:06 smithi196 ceph-mon[76349]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:08 smithi184 ceph-mon[100622]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:08 smithi184 ceph-mon[102844]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:08 smithi196 ceph-mon[76349]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:10.559 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:51:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:10 smithi196 ceph-mon[76349]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:10 smithi184 ceph-mon[100622]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:10 smithi184 ceph-mon[102844]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:11.081 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 6h - - 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 6h - - 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 11s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 9s ago 6h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 9s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 11s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 11s ago 6h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 9s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 11s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 11s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 9s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:51:11.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 11s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 11s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 11s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 11s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 9s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 9s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 9s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 9s ago 6h 69.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 9s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:51:11.083 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 9s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:51:11.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:51:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:51:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:51:11.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:51:11.668 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:51:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:51:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:51:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:51:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:51:11.670 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:51:11.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-mon[100622]: from='client.68409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:11.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2362693140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:11.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-mon[102844]: from='client.68409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:11.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2362693140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:11.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:11 smithi196 ceph-mon[76349]: from='client.68409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2362693140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:11.980 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:51:11.981 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:11.981 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:11.981 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:11.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:11.981 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:11.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:11.981 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:11.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:51:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[102844]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[102844]: from='client.68415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[102844]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/420129749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[100622]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[100622]: from='client.68415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[100622]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/420129749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:12 smithi196 ceph-mon[76349]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:12 smithi196 ceph-mon[76349]: from='client.68415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:12 smithi196 ceph-mon[76349]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/420129749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:13.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:13 smithi184 ceph-mon[100622]: from='client.78398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:13.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:13 smithi184 ceph-mon[102844]: from='client.78398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:13.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:13 smithi196 ceph-mon[76349]: from='client.78398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:14 smithi184 ceph-mon[100622]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:14 smithi184 ceph-mon[102844]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:14 smithi196 ceph-mon[76349]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:16 smithi184 ceph-mon[100622]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:16 smithi184 ceph-mon[102844]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:16.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:16 smithi196 ceph-mon[76349]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:18.766 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:18 smithi196 ceph-mon[76349]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:18 smithi184 ceph-mon[100622]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:18 smithi184 ceph-mon[102844]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[100622]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[102844]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:20 smithi196 ceph-mon[76349]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:21.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:21.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:21.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[102844]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[100622]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:22 smithi196 ceph-mon[76349]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:24 smithi184 ceph-mon[100622]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:24 smithi184 ceph-mon[102844]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:24 smithi196 ceph-mon[76349]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:26.811 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:26 smithi184 ceph-mon[100622]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:26.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:26 smithi184 ceph-mon[102844]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:26 smithi196 ceph-mon[76349]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:28 smithi196 ceph-mon[76349]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:28 smithi184 ceph-mon[102844]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:28 smithi184 ceph-mon[100622]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:30 smithi196 ceph-mon[76349]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:30 smithi184 ceph-mon[100622]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:30 smithi184 ceph-mon[102844]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:31.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:31.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:31.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:51:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:32 smithi196 ceph-mon[76349]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:32 smithi184 ceph-mon[100622]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:32 smithi184 ceph-mon[102844]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:34 smithi196 ceph-mon[76349]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:34 smithi184 ceph-mon[100622]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:34 smithi184 ceph-mon[102844]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:36 smithi196 ceph-mon[76349]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:36 smithi184 ceph-mon[100622]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:36 smithi184 ceph-mon[102844]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:38 smithi196 ceph-mon[76349]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:38 smithi184 ceph-mon[100622]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:38 smithi184 ceph-mon[102844]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:40 smithi196 ceph-mon[76349]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[100622]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[102844]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:41.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:41.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:41.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:42.277 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:51:42.278 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:51:42.805 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 2s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 2s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 2s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 2s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 2s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 2s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 2s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:51:42.806 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 2s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 2s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 2s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 1s ago 6h 69.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:51:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:51:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:42 smithi196 ceph-mon[76349]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[100622]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[102844]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:42.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:51:43.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:51:43.119 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:51:43.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:51:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:51:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:51:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:51:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:51:43.387 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:51:43.703 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:51:43.703 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:43.703 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:43.703 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:43.703 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:43.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[100622]: from='client.68445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[100622]: from='client.78416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/520044738' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:43.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[102844]: from='client.68445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[102844]: from='client.78416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/520044738' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:43 smithi196 ceph-mon[76349]: from='client.68445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:43 smithi196 ceph-mon[76349]: from='client.78416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/520044738' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:44 smithi196 ceph-mon[76349]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:44 smithi196 ceph-mon[76349]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:44 smithi196 ceph-mon[76349]: from='client.78434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2551190481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[100622]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[100622]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[100622]: from='client.78434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2551190481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[102844]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[102844]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[102844]: from='client.78434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2551190481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:45.782 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:45.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:46.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:46 smithi184 ceph-mon[102844]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:46.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:46 smithi184 ceph-mon[100622]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:46 smithi196 ceph-mon[76349]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:48 smithi196 ceph-mon[76349]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:48 smithi184 ceph-mon[100622]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:48 smithi184 ceph-mon[102844]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:50 smithi196 ceph-mon[76349]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:50 smithi184 ceph-mon[100622]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:50 smithi184 ceph-mon[102844]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:51.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:51.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:51:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:51:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:51:51.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:51:52.300 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:51:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:51:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:52 smithi196 ceph-mon[76349]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:52 smithi184 ceph-mon[100622]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:52 smithi184 ceph-mon[102844]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:54 smithi196 ceph-mon[76349]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:54 smithi184 ceph-mon[100622]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:54 smithi184 ceph-mon[102844]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:56 smithi196 ceph-mon[76349]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:56 smithi184 ceph-mon[100622]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:56 smithi184 ceph-mon[102844]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:58 smithi196 ceph-mon[76349]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:58 smithi184 ceph-mon[100622]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:58 smithi184 ceph-mon[102844]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:59.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:00.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:00.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:00 smithi196 ceph-mon[76349]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[100622]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[102844]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:01.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:01.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:01.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[100622]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[102844]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:02 smithi196 ceph-mon[76349]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:04 smithi184 ceph-mon[100622]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:04 smithi184 ceph-mon[102844]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:05.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:04 smithi196 ceph-mon[76349]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:06 smithi184 ceph-mon[100622]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:06 smithi184 ceph-mon[102844]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:07.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:06 smithi196 ceph-mon[76349]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:08 smithi184 ceph-mon[102844]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:08 smithi184 ceph-mon[100622]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:09.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:08 smithi196 ceph-mon[76349]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:10 smithi184 ceph-mon[102844]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:10 smithi184 ceph-mon[100622]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:11.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:10 smithi196 ceph-mon[76349]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:11.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:11.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:11.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:52:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:12 smithi184 ceph-mon[102844]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:12 smithi184 ceph-mon[100622]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:12 smithi196 ceph-mon[76349]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:13.999 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:52:14.543 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 14s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 14s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 12s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 14s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:52:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 14s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 14s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 14s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 14s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 12s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 12s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 12s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 12s ago 6h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 12s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:52:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 12s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:52:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:14 smithi184 ceph-mon[100622]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:14 smithi184 ceph-mon[102844]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:14.858 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:52:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:52:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:52:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:52:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:52:14.860 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:52:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:52:15.130 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:52:15.130 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:52:15.130 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:52:15.130 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:52:15.130 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:52:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:14 smithi196 ceph-mon[76349]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:15.437 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:52:15.438 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:15.438 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.68481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.78452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.78458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1075358877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2061019603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.68481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.78452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.78458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1075358877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2061019603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.68481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.78452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.78458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1075358877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2061019603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:16 smithi184 ceph-mon[102844]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:16 smithi184 ceph-mon[102844]: from='client.78470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:16 smithi184 ceph-mon[100622]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:16 smithi184 ceph-mon[100622]: from='client.78470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:17.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:16 smithi196 ceph-mon[76349]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:17.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:16 smithi196 ceph-mon[76349]: from='client.78470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:17.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:17 smithi196 ceph-mon[76349]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:18.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:17 smithi184 ceph-mon[102844]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:17 smithi184 ceph-mon[100622]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:20.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:19 smithi196 ceph-mon[76349]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:20.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:19 smithi184 ceph-mon[100622]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:20.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:19 smithi184 ceph-mon[102844]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:21.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:21.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:21.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:21.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:22.291 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:52:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[100622]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[102844]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:22 smithi196 ceph-mon[76349]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:24 smithi184 ceph-mon[100622]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:24 smithi184 ceph-mon[102844]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:24 smithi196 ceph-mon[76349]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:26 smithi196 ceph-mon[76349]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:26.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:26 smithi184 ceph-mon[100622]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:26.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:26 smithi184 ceph-mon[102844]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:28.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:28 smithi184 ceph-mon[100622]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:28.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:28 smithi184 ceph-mon[102844]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:28 smithi196 ceph-mon[76349]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:30 smithi196 ceph-mon[76349]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:30 smithi184 ceph-mon[100622]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:30 smithi184 ceph-mon[102844]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:31.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:31.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:31.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:52:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:32 smithi196 ceph-mon[76349]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:32 smithi184 ceph-mon[100622]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:32 smithi184 ceph-mon[102844]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:34 smithi196 ceph-mon[76349]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:34 smithi184 ceph-mon[102844]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:34 smithi184 ceph-mon[100622]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:36 smithi196 ceph-mon[76349]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:36 smithi184 ceph-mon[100622]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:36 smithi184 ceph-mon[102844]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:38 smithi196 ceph-mon[76349]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:38 smithi184 ceph-mon[100622]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:38 smithi184 ceph-mon[102844]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:40 smithi196 ceph-mon[76349]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:40 smithi184 ceph-mon[100622]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:40 smithi184 ceph-mon[102844]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:41.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:41.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:41.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:52:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[100622]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[102844]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:42 smithi196 ceph-mon[76349]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:52:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:44 smithi184 ceph-mon[102844]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:44 smithi184 ceph-mon[100622]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:44 smithi196 ceph-mon[76349]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:52:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:45.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:45.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:45.731 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:52:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:52:46.256 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 5s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 5s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 4s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 5s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 5s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 5s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 5s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 5s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 4s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 4s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 4s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 4s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:52:46.257 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 4s ago 6h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:52:46.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:52:46.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 4s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:52:46.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:52:46.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 4s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:52:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:46 smithi184 ceph-mon[100622]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:46 smithi184 ceph-mon[102844]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:46.573 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:52:46.574 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:52:46.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:46 smithi196 ceph-mon[76349]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:52:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:52:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:52:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:52:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:52:46.844 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:52:47.153 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:52:47.153 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:47.153 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:47.153 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:47.153 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:47 smithi196 ceph-mon[76349]: from='client.78482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:47 smithi196 ceph-mon[76349]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:47 smithi196 ceph-mon[76349]: from='client.78494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1227989299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1244049033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:47.743 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[100622]: from='client.78482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[100622]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[100622]: from='client.78494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1227989299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1244049033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[102844]: from='client.78482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[102844]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[102844]: from='client.78494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1227989299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:47.744 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1244049033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:48.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:48 smithi184 ceph-mon[100622]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:48.752 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:48 smithi184 ceph-mon[100622]: from='client.78506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:48.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:48 smithi184 ceph-mon[102844]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:48.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:48 smithi184 ceph-mon[102844]: from='client.78506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:48 smithi196 ceph-mon[76349]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:48 smithi196 ceph-mon[76349]: from='client.78506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:50 smithi184 ceph-mon[100622]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:50 smithi184 ceph-mon[102844]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:50 smithi196 ceph-mon[76349]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:51.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:51.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:52:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:52:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:52:51.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:52:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:52:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:52:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:52 smithi184 ceph-mon[100622]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:52 smithi184 ceph-mon[102844]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:52 smithi196 ceph-mon[76349]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:54 smithi184 ceph-mon[100622]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:54 smithi184 ceph-mon[102844]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:54 smithi196 ceph-mon[76349]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:56 smithi184 ceph-mon[100622]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:56 smithi184 ceph-mon[102844]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:56 smithi196 ceph-mon[76349]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:58 smithi184 ceph-mon[100622]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:58 smithi184 ceph-mon[102844]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:58 smithi196 ceph-mon[76349]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:52:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:52:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:52:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:00 smithi184 ceph-mon[102844]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:00 smithi184 ceph-mon[100622]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:00.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:00 smithi196 ceph-mon[76349]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:01.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.728 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:01.729 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:01.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:01.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:01.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[100622]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[102844]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:02 smithi196 ceph-mon[76349]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:04 smithi184 ceph-mon[100622]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:04 smithi184 ceph-mon[102844]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:04 smithi196 ceph-mon[76349]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:06 smithi196 ceph-mon[76349]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:06 smithi184 ceph-mon[100622]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:06 smithi184 ceph-mon[102844]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:08 smithi196 ceph-mon[76349]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:09.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:08 smithi184 ceph-mon[100622]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:09.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:08 smithi184 ceph-mon[102844]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:10.816 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:10 smithi184 ceph-mon[100622]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:10.816 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:10 smithi184 ceph-mon[102844]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:10 smithi196 ceph-mon[76349]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:11.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:11.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:11.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:53:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:12 smithi196 ceph-mon[76349]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:12 smithi184 ceph-mon[100622]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:12 smithi184 ceph-mon[102844]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:14 smithi196 ceph-mon[76349]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:14 smithi184 ceph-mon[100622]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:14 smithi184 ceph-mon[102844]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:16 smithi196 ceph-mon[76349]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:16 smithi184 ceph-mon[100622]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:16 smithi184 ceph-mon[102844]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:17.448 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:53:17.973 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:53:17.973 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 6h - - 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 17s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 17s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 17s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 17s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 15s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 17s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 17s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 17s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:53:17.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 17s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 17s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 17s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 15s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 15s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 15s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 15s ago 6h 70.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 15s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:53:17.975 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 15s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:53:18.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:53:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:53:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:53:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:53:18.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:53:18.549 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:53:18.549 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:53:18.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:53:18.862 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:53:18.862 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:18.862 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:18.862 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[102844]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[102844]: from='client.78518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2782385409' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[100622]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[100622]: from='client.78518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:18.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2782385409' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:18 smithi196 ceph-mon[76349]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:18 smithi196 ceph-mon[76349]: from='client.78518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2782385409' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:19 smithi196 ceph-mon[76349]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:19.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:19 smithi196 ceph-mon[76349]: from='client.78530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:19 smithi196 ceph-mon[76349]: from='client.78542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4217054453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[102844]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[102844]: from='client.78530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[102844]: from='client.78542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4217054453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[100622]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[100622]: from='client.78530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[100622]: from='client.78542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4217054453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:20.853 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:20 smithi184 ceph-mon[100622]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:20.853 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:20 smithi184 ceph-mon[102844]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:20 smithi196 ceph-mon[76349]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:21.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:21.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:21.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[100622]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[102844]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:22 smithi196 ceph-mon[76349]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:23 smithi196 ceph-mon[76349]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:23 smithi184 ceph-mon[100622]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:23 smithi184 ceph-mon[102844]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:24 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:25.058 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:24 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:24 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:25 smithi184 ceph-mon[100622]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:25 smithi184 ceph-mon[102844]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:25 smithi196 ceph-mon[76349]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:27 smithi196 ceph-mon[76349]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:27 smithi184 ceph-mon[100622]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:28.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:27 smithi184 ceph-mon[102844]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:30.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:29 smithi184 ceph-mon[100622]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:30.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:30.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:29 smithi184 ceph-mon[102844]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:30.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:29 smithi196 ceph-mon[76349]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:31 smithi184 ceph-mon[100622]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:31 smithi184 ceph-mon[102844]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:31.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:31.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:31.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:31 smithi196 ceph-mon[76349]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:53:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:33 smithi196 ceph-mon[76349]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:34.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:33 smithi184 ceph-mon[100622]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:34.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:33 smithi184 ceph-mon[102844]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:35 smithi196 ceph-mon[76349]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:35 smithi184 ceph-mon[100622]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:36.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:35 smithi184 ceph-mon[102844]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:37 smithi196 ceph-mon[76349]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:37 smithi184 ceph-mon[100622]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:37 smithi184 ceph-mon[102844]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:40.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:39 smithi196 ceph-mon[76349]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:39 smithi184 ceph-mon[100622]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:39 smithi184 ceph-mon[102844]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:41.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[102844]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:41.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[100622]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:41.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:41.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:41.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:41.986 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:41.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:42.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:42.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:42.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:41 smithi196 ceph-mon[76349]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:42.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:43.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:53:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:44 smithi196 ceph-mon[76349]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:44 smithi184 ceph-mon[100622]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:44 smithi184 ceph-mon[102844]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:45.883 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:46 smithi184 ceph-mon[102844]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:46 smithi184 ceph-mon[100622]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:46 smithi196 ceph-mon[76349]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:48 smithi184 ceph-mon[100622]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:48 smithi184 ceph-mon[102844]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:48 smithi196 ceph-mon[76349]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:49.158 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:53:49.710 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 6h - - 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 7s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 7s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 8s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 8s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 7s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 8s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 7s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 8s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 8s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:53:49.711 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 8s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 8s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 7s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 7s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 7s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 7s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 7s ago 6h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 7s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:53:49.712 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 7s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:53:50.025 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:53:50.025 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:53:50.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:53:50.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:53:50.027 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:53:50.299 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:53:50.300 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:53:50.301 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:53:50.301 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:53:50.618 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:53:50.619 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:50.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:50.619 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:50.619 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:50.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[100622]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:50.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[100622]: from='client.78554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.619 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[100622]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4052082812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:50.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[102844]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:50.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[102844]: from='client.78554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[102844]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.620 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4052082812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:50 smithi196 ceph-mon[76349]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:50 smithi196 ceph-mon[76349]: from='client.78554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:50 smithi196 ceph-mon[76349]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4052082812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[100622]: from='client.78566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[100622]: from='client.68613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1874731922' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[102844]: from='client.78566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[102844]: from='client.68613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.572 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1874731922' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:51 smithi196 ceph-mon[76349]: from='client.78566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:51 smithi196 ceph-mon[76349]: from='client.68613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1874731922' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:51.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:51.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:51.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:51.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:53:51.967 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:53:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:53:51.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:53:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:53:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:53:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:52 smithi184 ceph-mon[100622]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:52 smithi184 ceph-mon[102844]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:52 smithi196 ceph-mon[76349]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:54 smithi184 ceph-mon[100622]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:54 smithi184 ceph-mon[102844]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:54 smithi196 ceph-mon[76349]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:56 smithi184 ceph-mon[100622]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:56 smithi184 ceph-mon[102844]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:56 smithi196 ceph-mon[76349]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:58 smithi184 ceph-mon[100622]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:58 smithi184 ceph-mon[102844]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:58 smithi196 ceph-mon[76349]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:53:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:53:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:53:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:00 smithi184 ceph-mon[100622]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:00 smithi184 ceph-mon[102844]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:00 smithi196 ceph-mon[76349]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:02.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[102844]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:01.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:01.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:01.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[100622]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:54:02.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:01 smithi196 ceph-mon[76349]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:02.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:02.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:03.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:04.781 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:04 smithi196 ceph-mon[76349]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:04 smithi184 ceph-mon[102844]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:04 smithi184 ceph-mon[100622]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:06 smithi196 ceph-mon[76349]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:06 smithi184 ceph-mon[102844]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:06 smithi184 ceph-mon[100622]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:08 smithi196 ceph-mon[76349]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:54:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:08 smithi184 ceph-mon[102844]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:54:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:08 smithi184 ceph-mon[100622]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:54:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:10 smithi184 ceph-mon[100622]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:54:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:10 smithi184 ceph-mon[102844]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:54:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:10 smithi196 ceph-mon[76349]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:54:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:11.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:11.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:11.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:11.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:12.289 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:54:12.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:12 smithi184 ceph-mon[100622]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:12.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:12 smithi184 ceph-mon[102844]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:12 smithi196 ceph-mon[76349]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:14 smithi196 ceph-mon[76349]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:14 smithi184 ceph-mon[100622]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:14 smithi184 ceph-mon[102844]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:16 smithi196 ceph-mon[76349]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:54:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:16 smithi184 ceph-mon[100622]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:54:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:16 smithi184 ceph-mon[102844]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 404 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T01:54:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:18 smithi196 ceph-mon[76349]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 408 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T01:54:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:18 smithi184 ceph-mon[100622]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 408 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T01:54:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:18 smithi184 ceph-mon[102844]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 408 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T01:54:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:20 smithi196 ceph-mon[76349]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:54:20.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:20 smithi184 ceph-mon[100622]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:54:20.912 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:20 smithi184 ceph-mon[102844]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:54:20.913 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 6h - - 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 0s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:54:21.452 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 0s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (5h) 0s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 0s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 18s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 0s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 0s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 0s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 0s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 0s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 0s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 18s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 18s ago 6h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 18s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:54:21.453 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (5h) 18s ago 6h 70.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:54:21.454 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:54:21.454 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 18s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:54:21.454 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:54:21.454 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 18s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:54:21.764 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:54:21.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:54:21.765 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:54:22.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:21.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:22.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:21.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:22.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:21.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:22.034 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:54:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:54:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:54:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:54:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:54:22.035 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='client.68625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='client.68631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2736945502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='client.68625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='client.68631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2736945502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:22.348 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:54:22.349 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:22.349 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:22.349 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:22.349 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:22.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='client.68625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='client.68631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2736945502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:23.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:23 smithi196 ceph-mon[76349]: from='client.68649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:23.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3084379413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:23.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[100622]: from='client.68649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3084379413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[102844]: from='client.68649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3084379413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:24.717 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:24 smithi196 ceph-mon[76349]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:24 smithi184 ceph-mon[100622]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:24 smithi184 ceph-mon[102844]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:54:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:26 smithi184 ceph-mon[100622]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T01:54:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:26 smithi184 ceph-mon[102844]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T01:54:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:26 smithi196 ceph-mon[76349]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T01:54:28.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:27 smithi184 ceph-mon[100622]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:54:28.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:27 smithi184 ceph-mon[102844]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:54:28.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:27 smithi196 ceph-mon[76349]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:54:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:29 smithi196 ceph-mon[76349]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:54:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:29 smithi184 ceph-mon[100622]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:54:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:29 smithi184 ceph-mon[102844]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:54:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:31 smithi184 ceph-mon[100622]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:31 smithi184 ceph-mon[102844]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:31.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:31.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:31.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:32.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:31 smithi196 ceph-mon[76349]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:54:34.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:33 smithi196 ceph-mon[76349]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:54:34.244 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:33 smithi184 ceph-mon[100622]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:54:34.244 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:33 smithi184 ceph-mon[102844]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:54:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:34 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:34 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:34 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:35 smithi196 ceph-mon[76349]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:54:36.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:35 smithi184 ceph-mon[100622]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:54:36.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:35 smithi184 ceph-mon[102844]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T01:54:38.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:37 smithi196 ceph-mon[76349]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:54:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:37 smithi184 ceph-mon[100622]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:54:38.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:37 smithi184 ceph-mon[102844]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T01:54:40.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:39 smithi184 ceph-mon[100622]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:54:40.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:39 smithi184 ceph-mon[102844]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:54:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:39 smithi196 ceph-mon[76349]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:54:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:41.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:41.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:41.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:54:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[102844]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[100622]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:42 smithi196 ceph-mon[76349]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:54:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.011 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[100622]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:43 smithi184 ceph-mon[102844]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:54:44.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:54:44.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:43 smithi196 ceph-mon[76349]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:54:45.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:45.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:46.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:45 smithi196 ceph-mon[76349]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:54:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:45 smithi184 ceph-mon[100622]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:54:46.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:45 smithi184 ceph-mon[102844]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:54:48.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:47 smithi196 ceph-mon[76349]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:54:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:47 smithi184 ceph-mon[100622]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:54:48.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:47 smithi184 ceph-mon[102844]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:54:50.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:49 smithi196 ceph-mon[76349]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:54:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:49 smithi184 ceph-mon[100622]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:54:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:49 smithi184 ceph-mon[102844]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T01:54:52.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:51 smithi184 ceph-mon[100622]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:52.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:51 smithi184 ceph-mon[102844]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:51.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:51.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:54:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:54:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:54:51.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:54:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:51 smithi196 ceph-mon[76349]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:54:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:54:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:54:52.649 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:54:52.976 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:52 smithi184 ceph-mon[100622]: from='client.78626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:52.976 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:52 smithi184 ceph-mon[102844]: from='client.78626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:52 smithi196 ceph-mon[76349]: from='client.78626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:53.205 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 6h - - 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 10s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (5h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (5h) 10s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 12s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 10s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 12s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 10s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 12s ago 6h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:54:53.206 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 6h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 10s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 10s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 10s ago 6h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 10s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 6h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 10s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:54:53.207 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 10s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:54:53.528 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:54:53.528 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:54:53.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:54:53.529 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:54:53.811 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:54:53.812 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:54:54.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[102844]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:54.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[102844]: from='client.78632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[102844]: from='client.78638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2601059060' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:54.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[100622]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:54.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[100622]: from='client.78632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[100622]: from='client.78638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.022 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2601059060' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:54.127 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:54:54.127 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:54.127 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:54.127 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:54.127 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:53 smithi196 ceph-mon[76349]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:53 smithi196 ceph-mon[76349]: from='client.78632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:53 smithi196 ceph-mon[76349]: from='client.78638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2601059060' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[100622]: from='client.68685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2355964281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[102844]: from='client.68685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2355964281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:54 smithi196 ceph-mon[76349]: from='client.68685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2355964281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:55 smithi184 ceph-mon[100622]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:55 smithi184 ceph-mon[102844]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:55 smithi196 ceph-mon[76349]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:58.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:57 smithi196 ceph-mon[76349]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:57 smithi184 ceph-mon[100622]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:57 smithi184 ceph-mon[102844]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:59 smithi184 ceph-mon[102844]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:54:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:59 smithi184 ceph-mon[100622]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:54:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:59 smithi196 ceph-mon[76349]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:54:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:01.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:01.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:01.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:55:02.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:02 smithi196 ceph-mon[76349]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[100622]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[102844]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[100622]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[102844]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:03 smithi196 ceph-mon[76349]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:05.075 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:04 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:04 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:04 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:05 smithi196 ceph-mon[76349]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:05 smithi184 ceph-mon[100622]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:06.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:05 smithi184 ceph-mon[102844]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:07 smithi184 ceph-mon[100622]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:08.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:07 smithi184 ceph-mon[102844]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:08.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:07 smithi196 ceph-mon[76349]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:09 smithi184 ceph-mon[100622]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:10.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:09 smithi184 ceph-mon[102844]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:10.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:09 smithi196 ceph-mon[76349]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:11 smithi184 ceph-mon[100622]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:11 smithi184 ceph-mon[102844]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:11.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:11.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:11.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:55:12.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:11 smithi196 ceph-mon[76349]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:14.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:13 smithi184 ceph-mon[100622]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:14.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:13 smithi184 ceph-mon[102844]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:14.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:13 smithi196 ceph-mon[76349]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:15.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:15.217 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:15.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:15.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:15.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:16.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:15 smithi196 ceph-mon[76349]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:16.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:15 smithi184 ceph-mon[100622]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:16.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:15 smithi184 ceph-mon[102844]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:17 smithi184 ceph-mon[100622]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:17 smithi184 ceph-mon[102844]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:18.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:17 smithi196 ceph-mon[76349]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:20.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:19 smithi184 ceph-mon[100622]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:20.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:19 smithi184 ceph-mon[102844]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:20.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:19 smithi196 ceph-mon[76349]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:21.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:21.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:21.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:55:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[102844]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[100622]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:22 smithi196 ceph-mon[76349]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:24 smithi196 ceph-mon[76349]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[100622]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:24.416 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[102844]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:24.417 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:55:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:55:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 6h - - 2024-09-21T01:55:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T01:55:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 3s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:55:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 3s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 1s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 3s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 3s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:55:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 3s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 3s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 3s ago 6h 81.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 3s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 1s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 1s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 1s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 1s ago 6h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:55:24.947 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:55:24.948 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 1s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:55:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:55:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:55:25.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[100622]: from='client.68697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[100622]: from='client.78668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[102844]: from='client.68697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:25 smithi184 ceph-mon[102844]: from='client.78668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:25.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:25 smithi196 ceph-mon[76349]: from='client.68697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:25 smithi196 ceph-mon[76349]: from='client.78668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:55:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:55:25.538 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:55:25.850 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:55:25.850 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:25.850 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:25.850 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:25.851 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:26.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:26 smithi196 ceph-mon[76349]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:26 smithi196 ceph-mon[76349]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3497518813' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:26 smithi196 ceph-mon[76349]: from='client.78686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1224396748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[100622]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[100622]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3497518813' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[100622]: from='client.78686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1224396748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[102844]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[102844]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3497518813' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[102844]: from='client.78686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1224396748' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:28.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:28 smithi196 ceph-mon[76349]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:28 smithi184 ceph-mon[100622]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:28 smithi184 ceph-mon[102844]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:30.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:30 smithi196 ceph-mon[76349]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:30.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:30 smithi184 ceph-mon[100622]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:30 smithi184 ceph-mon[102844]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:31.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:31.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:31.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:32.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:32 smithi184 ceph-mon[100622]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:32 smithi184 ceph-mon[102844]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:55:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:32 smithi196 ceph-mon[76349]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:34.346 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:34 smithi184 ceph-mon[100622]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:34.346 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:34 smithi184 ceph-mon[102844]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:34 smithi196 ceph-mon[76349]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:35.214 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:35.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:35.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:36.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:36 smithi196 ceph-mon[76349]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:36 smithi184 ceph-mon[102844]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:36 smithi184 ceph-mon[100622]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:38.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:38 smithi196 ceph-mon[76349]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:38 smithi184 ceph-mon[100622]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:38.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:38 smithi184 ceph-mon[102844]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:40 smithi196 ceph-mon[76349]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:40 smithi184 ceph-mon[100622]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:40 smithi184 ceph-mon[102844]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:41.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:41.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:41.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:55:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:42 smithi196 ceph-mon[76349]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[100622]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[102844]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[102844]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[100622]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:44 smithi196 ceph-mon[76349]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:55:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:46 smithi184 ceph-mon[102844]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:46 smithi184 ceph-mon[100622]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:46 smithi196 ceph-mon[76349]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:48.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:48 smithi196 ceph-mon[76349]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:48 smithi184 ceph-mon[102844]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:48 smithi184 ceph-mon[100622]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:50 smithi196 ceph-mon[76349]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:50.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:50 smithi184 ceph-mon[102844]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:50 smithi184 ceph-mon[100622]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:51.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:51.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:55:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:55:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:55:51.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:55:52.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:52 smithi184 ceph-mon[100622]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:55:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:55:52.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:52 smithi184 ceph-mon[102844]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:52 smithi196 ceph-mon[76349]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:54 smithi184 ceph-mon[100622]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:54 smithi184 ceph-mon[102844]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:54 smithi196 ceph-mon[76349]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:55.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:55.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:56.156 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:55:56.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:56 smithi184 ceph-mon[100622]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:56.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:56 smithi184 ceph-mon[102844]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:56 smithi196 ceph-mon[76349]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:56.714 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 6h - - 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 13s ago 6h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (5h) 15s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 13s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (5h) 15s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 13s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 15s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 15s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:55:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 15s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 13s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 13s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 13s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 13s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 6h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:55:56.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 13s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:55:57.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:55:57.036 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[102844]: from='client.78698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[102844]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/959085943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[100622]: from='client.78698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[100622]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/959085943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:57.316 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:55:57.316 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:55:57.316 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:55:57.317 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:55:57.624 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:55:57.625 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:57.625 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:57.625 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:57.625 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:57 smithi196 ceph-mon[76349]: from='client.78698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:57 smithi196 ceph-mon[76349]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/959085943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[100622]: from='client.78710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[100622]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[100622]: from='client.78722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/448546082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[102844]: from='client.78710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[102844]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[102844]: from='client.78722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/448546082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:58.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:58 smithi196 ceph-mon[76349]: from='client.78710 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:58 smithi196 ceph-mon[76349]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:58 smithi196 ceph-mon[76349]: from='client.78722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/448546082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:59.513 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:00 smithi196 ceph-mon[76349]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:00 smithi184 ceph-mon[100622]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:00 smithi184 ceph-mon[102844]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:02.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:01.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:01.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:01.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:56:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[100622]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[102844]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:02 smithi196 ceph-mon[76349]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[100622]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:04.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[102844]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:04.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.555 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:04 smithi196 ceph-mon[76349]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:04.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:06 smithi196 ceph-mon[76349]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:06.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:06 smithi184 ceph-mon[100622]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:06.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:06 smithi184 ceph-mon[102844]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:08.413 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:08 smithi196 ceph-mon[76349]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:08 smithi184 ceph-mon[100622]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:08 smithi184 ceph-mon[102844]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:10 smithi196 ceph-mon[76349]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:10 smithi184 ceph-mon[100622]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:10 smithi184 ceph-mon[102844]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:11.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:11.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:11.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T01:56:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:12 smithi196 ceph-mon[76349]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:12 smithi184 ceph-mon[102844]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:12 smithi184 ceph-mon[100622]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:14 smithi184 ceph-mon[102844]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:14.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:14 smithi184 ceph-mon[100622]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:14 smithi196 ceph-mon[76349]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:15.476 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:15.476 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:16 smithi184 ceph-mon[100622]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:16 smithi184 ceph-mon[102844]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:16 smithi196 ceph-mon[76349]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:18.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:18 smithi196 ceph-mon[76349]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:56:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:18 smithi184 ceph-mon[100622]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:56:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:18 smithi184 ceph-mon[102844]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:56:20.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:20 smithi196 ceph-mon[76349]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:20 smithi184 ceph-mon[100622]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:20 smithi184 ceph-mon[102844]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:21.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:21.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:21.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T01:56:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:22 smithi196 ceph-mon[76349]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[100622]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[102844]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[102844]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[100622]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:24 smithi196 ceph-mon[76349]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:26 smithi184 ceph-mon[100622]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:26 smithi184 ceph-mon[102844]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:26 smithi196 ceph-mon[76349]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:27.919 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:56:28.450 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 6h - - 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 6s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 6s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 6s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 6s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (5h) 5s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 6s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:56:28.451 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 6s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 6s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 6s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 6s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 6s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:56:28.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:56:28.453 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:56:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:28 smithi196 ceph-mon[76349]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:28.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:28 smithi184 ceph-mon[102844]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:56:28.766 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:56:28.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:56:28.767 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:28 smithi184 ceph-mon[100622]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:56:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:56:29.035 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:56:29.345 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:56:29.345 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:29.345 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:29.345 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:29.345 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='client.78734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='client.68775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='client.78746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2113483532' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3091907027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='client.78734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='client.68775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='client.78746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2113483532' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3091907027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='client.78734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='client.68775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='client.78746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2113483532' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3091907027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:30 smithi196 ceph-mon[76349]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:30.870 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:30 smithi196 ceph-mon[76349]: from='client.78758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:30.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:30 smithi184 ceph-mon[100622]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:30.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:30 smithi184 ceph-mon[100622]: from='client.78758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:30.871 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:30 smithi184 ceph-mon[102844]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:30.872 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:30 smithi184 ceph-mon[102844]: from='client.78758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:31.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:31.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:31.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:56:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:32 smithi184 ceph-mon[100622]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:32 smithi184 ceph-mon[102844]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:32 smithi196 ceph-mon[76349]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:34 smithi184 ceph-mon[100622]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:34 smithi184 ceph-mon[102844]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:34 smithi196 ceph-mon[76349]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:35.701 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:35.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:36 smithi184 ceph-mon[100622]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:36 smithi184 ceph-mon[102844]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:36 smithi196 ceph-mon[76349]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:38 smithi184 ceph-mon[100622]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:38 smithi184 ceph-mon[102844]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:38 smithi196 ceph-mon[76349]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:40 smithi196 ceph-mon[76349]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:40 smithi184 ceph-mon[100622]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:40 smithi184 ceph-mon[102844]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:41.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:41.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:41.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[100622]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[102844]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:42 smithi196 ceph-mon[76349]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:44 smithi196 ceph-mon[76349]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[100622]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[102844]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:56:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:45.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:46 smithi196 ceph-mon[76349]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:46 smithi184 ceph-mon[100622]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:46 smithi184 ceph-mon[102844]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:48 smithi196 ceph-mon[76349]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:48 smithi184 ceph-mon[100622]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:48 smithi184 ceph-mon[102844]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:50 smithi196 ceph-mon[76349]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:50 smithi184 ceph-mon[100622]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:50 smithi184 ceph-mon[102844]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:51.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:51.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:56:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:56:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:56:51.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:56:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:56:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:56:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:52 smithi196 ceph-mon[76349]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:52 smithi184 ceph-mon[100622]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:52 smithi184 ceph-mon[102844]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:54 smithi196 ceph-mon[76349]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:54 smithi184 ceph-mon[100622]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:54 smithi184 ceph-mon[102844]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:55.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:56.028 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:56.029 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:56.719 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:56 smithi184 ceph-mon[100622]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:56.725 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:56 smithi184 ceph-mon[102844]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:56 smithi196 ceph-mon[76349]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:58 smithi196 ceph-mon[76349]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:58 smithi184 ceph-mon[100622]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:58 smithi184 ceph-mon[102844]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.645 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:56:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:00.175 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 6h - - 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 16s ago 6h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 16s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 16s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 16s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:57:00.176 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 18s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 18s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 18s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 18s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 16s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 16s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 16s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 16s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 16s ago 6h 71.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:57:00.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 16s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:57:00.487 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:57:00.487 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:57:00.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:57:00.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:57:00.489 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:57:00.760 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:57:00.761 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:57:00.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[100622]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[100622]: from='client.68805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3152609306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[102844]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[102844]: from='client.68805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:00.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3152609306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:00 smithi196 ceph-mon[76349]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:00 smithi196 ceph-mon[76349]: from='client.68805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3152609306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:01.077 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:57:01.077 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:01.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:01.077 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:01.077 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:01.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:01 smithi196 ceph-mon[76349]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:01.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:01 smithi196 ceph-mon[76349]: from='client.78782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:01.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2912732117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[102844]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[102844]: from='client.78782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2912732117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[100622]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[100622]: from='client.78782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2912732117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:02.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:01.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:02.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:01.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:02.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:01.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[100622]: from='client.68829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[100622]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[102844]: from='client.68829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[102844]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:02 smithi196 ceph-mon[76349]: from='client.68829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:02 smithi196 ceph-mon[76349]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[100622]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[102844]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:04 smithi196 ceph-mon[76349]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:06.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:06 smithi184 ceph-mon[100622]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:06 smithi184 ceph-mon[102844]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:06 smithi196 ceph-mon[76349]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:08 smithi196 ceph-mon[76349]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:08 smithi184 ceph-mon[100622]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:08 smithi184 ceph-mon[102844]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:10 smithi184 ceph-mon[100622]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:10 smithi184 ceph-mon[102844]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:11.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:10 smithi196 ceph-mon[76349]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:11.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:11.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:11.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:57:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:12 smithi184 ceph-mon[100622]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:12 smithi184 ceph-mon[102844]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:12 smithi196 ceph-mon[76349]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:13 smithi184 ceph-mon[100622]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:13 smithi184 ceph-mon[102844]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:14.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:13 smithi196 ceph-mon[76349]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:14 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:14 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:14 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:16.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:15 smithi196 ceph-mon[76349]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:15 smithi184 ceph-mon[100622]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:15 smithi184 ceph-mon[102844]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:17 smithi184 ceph-mon[100622]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:17 smithi184 ceph-mon[102844]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:18.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:17 smithi196 ceph-mon[76349]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:20.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:19 smithi184 ceph-mon[100622]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:20.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:19 smithi184 ceph-mon[102844]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:20.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:19 smithi196 ceph-mon[76349]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:21 smithi184 ceph-mon[100622]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:21 smithi184 ceph-mon[102844]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:21.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:21.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:21.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:22.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:21 smithi196 ceph-mon[76349]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T01:57:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[100622]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[102844]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:24 smithi196 ceph-mon[76349]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:25.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:26 smithi184 ceph-mon[100622]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:26 smithi184 ceph-mon[102844]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:26 smithi196 ceph-mon[76349]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:28 smithi184 ceph-mon[100622]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:28.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:28 smithi184 ceph-mon[102844]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:28 smithi196 ceph-mon[76349]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:29.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:30.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:30 smithi196 ceph-mon[76349]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:30.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:30 smithi184 ceph-mon[100622]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:30 smithi184 ceph-mon[102844]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:31.369 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 9s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 97.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 9s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 9s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 9s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:57:31.897 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 9s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (5h) 9s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 9s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 9s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 9s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 8s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:57:31.898 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:57:32.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:31.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:32.001 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:31.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:32.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:31.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:57:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:57:32.219 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:57:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:57:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[102844]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[102844]: from='client.78806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[102844]: from='client.78812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4041569495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[100622]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[100622]: from='client.78806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[100622]: from='client.78812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4041569495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:32.499 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:57:32.500 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:57:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:32 smithi196 ceph-mon[76349]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:32 smithi196 ceph-mon[76349]: from='client.78806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:32 smithi196 ceph-mon[76349]: from='client.78812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4041569495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[100622]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[100622]: from='client.78830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2804722451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[102844]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[102844]: from='client.78830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2804722451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:33.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:33 smithi196 ceph-mon[76349]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:33 smithi196 ceph-mon[76349]: from='client.78830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2804722451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:34 smithi184 ceph-mon[100622]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:34 smithi184 ceph-mon[102844]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:34 smithi196 ceph-mon[76349]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:35.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:35.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:35.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:36 smithi184 ceph-mon[100622]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:36 smithi184 ceph-mon[102844]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:36 smithi196 ceph-mon[76349]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:38 smithi184 ceph-mon[100622]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:38.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:38 smithi184 ceph-mon[102844]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:38 smithi196 ceph-mon[76349]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:40 smithi184 ceph-mon[100622]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:40.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:40 smithi184 ceph-mon[102844]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:40.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:40 smithi196 ceph-mon[76349]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:41.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:41.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:41.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:42.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:57:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:42 smithi184 ceph-mon[100622]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:42.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:42 smithi184 ceph-mon[102844]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:42 smithi196 ceph-mon[76349]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[100622]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[102844]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:44.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:44 smithi196 ceph-mon[76349]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:57:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:45.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:46.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:46 smithi196 ceph-mon[76349]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:46 smithi184 ceph-mon[102844]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:46 smithi184 ceph-mon[100622]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:48 smithi196 ceph-mon[76349]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:48 smithi184 ceph-mon[100622]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:48 smithi184 ceph-mon[102844]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:50 smithi196 ceph-mon[76349]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:50 smithi184 ceph-mon[100622]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:50 smithi184 ceph-mon[102844]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:51.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:51.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:57:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:57:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:57:51.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:57:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:57:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T01:57:52.410 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:52 smithi196 ceph-mon[76349]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:52 smithi184 ceph-mon[100622]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:52 smithi184 ceph-mon[102844]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:54 smithi184 ceph-mon[100622]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:54 smithi184 ceph-mon[102844]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:54 smithi196 ceph-mon[76349]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:56 smithi196 ceph-mon[76349]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:56 smithi184 ceph-mon[100622]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:56 smithi184 ceph-mon[102844]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:58.462 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:58 smithi184 ceph-mon[100622]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:58.462 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:58 smithi184 ceph-mon[102844]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:58.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:58 smithi196 ceph-mon[76349]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:59.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:57:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:59.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:57:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:57:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:00 smithi184 ceph-mon[100622]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:00 smithi184 ceph-mon[102844]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:00 smithi196 ceph-mon[76349]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:01.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:01.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:02.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:01.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:58:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:02 smithi184 ceph-mon[100622]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:02 smithi184 ceph-mon[102844]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:02 smithi196 ceph-mon[76349]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:03.132 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:58:03.660 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 6h - - 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 19s ago 6h 92.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 19s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 19s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:58:03.661 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 19s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 1s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 1s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 1s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 19s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 19s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 19s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 19s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 19s ago 6h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:58:03.662 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:58:03.663 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 19s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:03.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:58:03.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:58:03.969 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:58:04.249 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:58:04.250 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:58:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:04.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='client.68877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='client.78854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/11520711' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='client.68877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='client.78854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/11520711' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:58:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:04.579 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:04.579 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:04.579 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='client.68877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='client.78854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:04.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/11520711' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:04.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[100622]: from='client.78866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/753767106' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[102844]: from='client.78866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/753767106' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:05 smithi196 ceph-mon[76349]: from='client.78866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/753767106' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:06 smithi184 ceph-mon[100622]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:06 smithi184 ceph-mon[102844]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:06 smithi196 ceph-mon[76349]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:08 smithi184 ceph-mon[100622]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:08 smithi184 ceph-mon[102844]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:08 smithi196 ceph-mon[76349]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:10.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:10 smithi196 ceph-mon[76349]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:10 smithi184 ceph-mon[100622]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:10 smithi184 ceph-mon[102844]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:11.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:11.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:12.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:11.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:58:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:12 smithi196 ceph-mon[76349]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:12 smithi184 ceph-mon[100622]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:12 smithi184 ceph-mon[102844]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 01:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T01:58:13.812144259Z level=info msg="Completed cleanup jobs" duration=12.686875ms 2024-09-21T01:58:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:14 smithi184 ceph-mon[100622]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:14 smithi184 ceph-mon[102844]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:14 smithi196 ceph-mon[76349]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:16.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:16 smithi196 ceph-mon[76349]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:16 smithi184 ceph-mon[102844]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:16 smithi184 ceph-mon[100622]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:18.788 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:18 smithi184 ceph-mon[100622]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:18.788 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:18 smithi184 ceph-mon[102844]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:18 smithi196 ceph-mon[76349]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:20 smithi184 ceph-mon[102844]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:20 smithi184 ceph-mon[100622]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:20 smithi196 ceph-mon[76349]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T01:58:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:21.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:21.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:21.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:22 smithi184 ceph-mon[100622]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:22 smithi184 ceph-mon[102844]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:22 smithi196 ceph-mon[76349]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:23.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:24 smithi196 ceph-mon[76349]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[100622]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[102844]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:26.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:26 smithi196 ceph-mon[76349]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:26 smithi184 ceph-mon[100622]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:26 smithi184 ceph-mon[102844]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:28 smithi196 ceph-mon[76349]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:28 smithi184 ceph-mon[100622]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:28 smithi184 ceph-mon[102844]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:30 smithi196 ceph-mon[76349]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:30 smithi184 ceph-mon[100622]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:30 smithi184 ceph-mon[102844]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:58:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:31.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:31.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:31.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:32 smithi184 ceph-mon[100622]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:32 smithi184 ceph-mon[102844]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:32 smithi196 ceph-mon[76349]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:58:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:34 smithi196 ceph-mon[76349]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:34 smithi184 ceph-mon[100622]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:34 smithi184 ceph-mon[102844]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:35.383 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:58:35.383 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T01:58:35.383 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T01:58:35.383 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:58:35.383 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 12s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 12s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (5h) 12s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 12s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 12s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 11s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:58:35.384 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 11s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:58:35.385 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:58:35.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:58:35.695 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:58:35.963 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:58:35.964 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:58:35.964 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:58:35.964 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:58:35.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:58:35.964 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:35.964 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:36.273 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:58:36.273 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:36.274 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:36.274 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:36.274 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: from='client.68913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: from='client.78890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1047020844' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1891227655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: from='client.68913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: from='client.78890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1047020844' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1891227655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: from='client.68913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: from='client.78890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1047020844' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1891227655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:38.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:37 smithi184 ceph-mon[100622]: from='client.68937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:38.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:37 smithi184 ceph-mon[102844]: from='client.68937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:37 smithi196 ceph-mon[76349]: from='client.68937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:38 smithi184 ceph-mon[100622]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:38 smithi184 ceph-mon[102844]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:38 smithi196 ceph-mon[76349]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:40.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:40 smithi184 ceph-mon[100622]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:40.776 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:40 smithi184 ceph-mon[102844]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:41.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:40 smithi196 ceph-mon[76349]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:41.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:41.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:41.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T01:58:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:42 smithi184 ceph-mon[100622]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:42 smithi184 ceph-mon[102844]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:42 smithi196 ceph-mon[76349]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:58:46.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:45 smithi196 ceph-mon[76349]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:45 smithi184 ceph-mon[100622]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:45 smithi184 ceph-mon[102844]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:47 smithi184 ceph-mon[100622]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:48.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:47 smithi184 ceph-mon[102844]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:48.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:47 smithi196 ceph-mon[76349]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:49 smithi184 ceph-mon[100622]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:50.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:49 smithi184 ceph-mon[102844]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:49 smithi196 ceph-mon[76349]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:52.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:51 smithi184 ceph-mon[100622]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:58:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T01:58:52.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:51 smithi184 ceph-mon[102844]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:51.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:51.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:58:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:58:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:58:51.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:58:52.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:51 smithi196 ceph-mon[76349]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:53 smithi196 ceph-mon[76349]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:54.233 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:53 smithi184 ceph-mon[100622]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:54.233 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:53 smithi184 ceph-mon[102844]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:54 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:54 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:55.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:54 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:55 smithi184 ceph-mon[100622]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:55 smithi184 ceph-mon[102844]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:55 smithi196 ceph-mon[76349]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:57 smithi184 ceph-mon[100622]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:57 smithi184 ceph-mon[102844]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:57 smithi196 ceph-mon[76349]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:00.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:59 smithi184 ceph-mon[100622]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:00.251 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:00.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:59 smithi184 ceph-mon[102844]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:00.252 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:59 smithi196 ceph-mon[76349]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:02.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:01 smithi184 ceph-mon[100622]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:59:02.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:01 smithi184 ceph-mon[102844]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:01.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:01.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:01.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:01 smithi196 ceph-mon[76349]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:03 smithi196 ceph-mon[76349]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:04.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[100622]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:04.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:04.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:03 smithi184 ceph-mon[102844]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:06.562 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:59:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:06 smithi184 ceph-mon[100622]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:06 smithi184 ceph-mon[102844]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:06 smithi196 ceph-mon[76349]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 6h - - 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 2s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 2s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:59:07.087 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 2s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (5h) 4s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 4s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 2s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 2s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 2s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 2s ago 6h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:59:07.088 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 2s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:59:07.397 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:59:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:59:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:59:07.398 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:59:07.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:07 smithi184 ceph-mon[100622]: from='client.78914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:07.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/470599890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:07.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:07 smithi184 ceph-mon[102844]: from='client.78914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:07.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/470599890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:59:07.670 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:59:07.671 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:59:07.671 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:59:07.671 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:59:07.671 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:59:07.671 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:59:07.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:07 smithi196 ceph-mon[76349]: from='client.78914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/470599890' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:07.979 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:59:07.979 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:07.979 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[100622]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[100622]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[100622]: from='client.78926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[100622]: from='client.78938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1175433509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[102844]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[102844]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[102844]: from='client.78926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[102844]: from='client.78938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1175433509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:08 smithi196 ceph-mon[76349]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:08 smithi196 ceph-mon[76349]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:08 smithi196 ceph-mon[76349]: from='client.78926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:08 smithi196 ceph-mon[76349]: from='client.78938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1175433509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:10 smithi184 ceph-mon[100622]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:10 smithi184 ceph-mon[102844]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:10 smithi196 ceph-mon[76349]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:59:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:11.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:11.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:11.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:12.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:12 smithi196 ceph-mon[76349]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:12 smithi184 ceph-mon[100622]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:12 smithi184 ceph-mon[102844]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:14.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:14 smithi196 ceph-mon[76349]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:14.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:14 smithi184 ceph-mon[102844]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:14 smithi184 ceph-mon[100622]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:16.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:16 smithi196 ceph-mon[76349]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:16 smithi184 ceph-mon[100622]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:16 smithi184 ceph-mon[102844]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:18 smithi196 ceph-mon[76349]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:18 smithi184 ceph-mon[100622]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:18 smithi184 ceph-mon[102844]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:20 smithi196 ceph-mon[76349]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:20.957 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:20 smithi184 ceph-mon[102844]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:20.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:20 smithi184 ceph-mon[100622]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:21.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:21.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:21.797 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:21.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:22.263 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T01:59:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:22 smithi196 ceph-mon[76349]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:22.978 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:22 smithi184 ceph-mon[100622]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:22.978 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:22 smithi184 ceph-mon[102844]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:24.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[100622]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:24.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.252 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.253 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[102844]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:24.253 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.253 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.253 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:23 smithi196 ceph-mon[76349]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:26 smithi196 ceph-mon[76349]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:26 smithi184 ceph-mon[100622]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:26 smithi184 ceph-mon[102844]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:28 smithi196 ceph-mon[76349]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:28 smithi184 ceph-mon[100622]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:28 smithi184 ceph-mon[102844]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:30 smithi196 ceph-mon[76349]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:30 smithi184 ceph-mon[100622]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:30 smithi184 ceph-mon[102844]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:59:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:31.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:31.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:31.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:32 smithi196 ceph-mon[76349]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:32 smithi184 ceph-mon[100622]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:32 smithi184 ceph-mon[102844]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:34.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:34 smithi196 ceph-mon[76349]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:34 smithi184 ceph-mon[100622]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:34 smithi184 ceph-mon[102844]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:35.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:36 smithi196 ceph-mon[76349]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:36 smithi184 ceph-mon[100622]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:36 smithi184 ceph-mon[102844]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:38.275 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T01:59:38.805 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T01:59:38.806 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (5h) 15s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 14s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 14s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 14s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 73.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T01:59:38.807 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T01:59:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:38 smithi196 ceph-mon[76349]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:38 smithi184 ceph-mon[100622]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:38 smithi184 ceph-mon[102844]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:39.117 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:59:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T01:59:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:59:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T01:59:39.119 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:59:39.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T01:59:39.387 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T01:59:39.698 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:59:39.698 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:39.699 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:39.699 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:39.699 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[100622]: from='client.68985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[100622]: from='client.78956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2064007545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[102844]: from='client.68985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[102844]: from='client.78956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.699 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2064007545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:39.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:39 smithi196 ceph-mon[76349]: from='client.68985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:39 smithi196 ceph-mon[76349]: from='client.78956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2064007545' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:40 smithi196 ceph-mon[76349]: from='client.78962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:40 smithi196 ceph-mon[76349]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:40 smithi196 ceph-mon[76349]: from='client.78974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/83153241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[100622]: from='client.78962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[100622]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[100622]: from='client.78974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/83153241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[102844]: from='client.78962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[102844]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[102844]: from='client.78974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/83153241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:41.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:41.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:41.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:59:42.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:42 smithi184 ceph-mon[100622]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:42.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:42 smithi184 ceph-mon[102844]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:42 smithi196 ceph-mon[76349]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:59:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[100622]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[102844]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:44 smithi196 ceph-mon[76349]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T01:59:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:46 smithi184 ceph-mon[102844]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:46 smithi184 ceph-mon[100622]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:47.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:46 smithi196 ceph-mon[76349]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:48 smithi184 ceph-mon[102844]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:48 smithi184 ceph-mon[100622]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:49.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:48 smithi196 ceph-mon[76349]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:50 smithi184 ceph-mon[102844]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:50 smithi184 ceph-mon[100622]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:50 smithi196 ceph-mon[76349]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 01:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:01:59:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T01:59:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:51.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:51.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T01:59:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 01:59:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T01:59:51.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T01:59:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:52 smithi184 ceph-mon[102844]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:52 smithi184 ceph-mon[100622]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:52 smithi196 ceph-mon[76349]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:54 smithi196 ceph-mon[76349]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:54 smithi184 ceph-mon[102844]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:54 smithi184 ceph-mon[100622]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:55 smithi184 ceph-mon[102844]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:55 smithi184 ceph-mon[100622]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:56.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:56.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:55 smithi196 ceph-mon[76349]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:57 smithi184 ceph-mon[102844]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:59:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:57 smithi184 ceph-mon[100622]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:59:58.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:57 smithi196 ceph-mon[76349]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:59 smithi184 ceph-mon[102844]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 01:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:59 smithi184 ceph-mon[100622]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 01:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T01:59:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:59 smithi196 ceph-mon[76349]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 01:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:00:02.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:01 smithi184 ceph-mon[102844]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:02.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:01 smithi184 ceph-mon[100622]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:01.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:01.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:01.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:02.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:01 smithi196 ceph-mon[76349]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:04.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[100622]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:04.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.307 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[102844]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:04.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.308 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:04 smithi196 ceph-mon[76349]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:00:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:06.819 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:06 smithi196 ceph-mon[76349]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:06.844 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:06 smithi184 ceph-mon[100622]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:06.844 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:06 smithi184 ceph-mon[102844]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:08 smithi196 ceph-mon[76349]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:08 smithi184 ceph-mon[100622]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:08 smithi184 ceph-mon[102844]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:09.992 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:00:10.521 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 5s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 5s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:00:10.522 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 5s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 5s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 73.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:00:10.523 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 5s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:00:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:10 smithi184 ceph-mon[100622]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:10 smithi184 ceph-mon[102844]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:00:10.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:00:10.835 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:00:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:10 smithi196 ceph-mon[76349]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:11.102 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:00:11.102 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:00:11.102 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:00:11.103 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:00:11.411 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:11.411 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:11.411 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:11.411 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:11.411 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[102844]: from='client.69021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[102844]: from='client.78992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[102844]: from='client.78998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3514010275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2305060230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[100622]: from='client.69021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[100622]: from='client.78992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[100622]: from='client.78998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3514010275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:11.763 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2305060230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:11 smithi196 ceph-mon[76349]: from='client.69021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:11 smithi196 ceph-mon[76349]: from='client.78992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:11 smithi196 ceph-mon[76349]: from='client.78998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3514010275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2305060230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:00:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:11.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:11.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:11.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:12 smithi196 ceph-mon[76349]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:12 smithi196 ceph-mon[76349]: from='client.79010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:12 smithi184 ceph-mon[100622]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:12 smithi184 ceph-mon[100622]: from='client.79010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:12 smithi184 ceph-mon[102844]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:12 smithi184 ceph-mon[102844]: from='client.79010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:14.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:14 smithi196 ceph-mon[76349]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:14.794 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:14 smithi184 ceph-mon[100622]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:14 smithi184 ceph-mon[102844]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:15.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:16 smithi196 ceph-mon[76349]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:16 smithi184 ceph-mon[100622]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:16 smithi184 ceph-mon[102844]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:18 smithi196 ceph-mon[76349]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:18 smithi184 ceph-mon[100622]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:18 smithi184 ceph-mon[102844]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:20 smithi196 ceph-mon[76349]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:20 smithi184 ceph-mon[100622]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:20 smithi184 ceph-mon[102844]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:00:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:21.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:21.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:21.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:22.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:22 smithi196 ceph-mon[76349]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:22.996 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:22 smithi184 ceph-mon[100622]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:22.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:22 smithi184 ceph-mon[102844]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[100622]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[102844]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:24 smithi196 ceph-mon[76349]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:26.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:26 smithi196 ceph-mon[76349]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:26 smithi184 ceph-mon[100622]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:26 smithi184 ceph-mon[102844]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:28.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:28 smithi196 ceph-mon[76349]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:28 smithi184 ceph-mon[100622]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:28 smithi184 ceph-mon[102844]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:30.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:30 smithi196 ceph-mon[76349]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:30 smithi184 ceph-mon[100622]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:30 smithi184 ceph-mon[102844]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:00:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:31.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:31.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:31.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:32 smithi196 ceph-mon[76349]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:32 smithi184 ceph-mon[100622]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:32 smithi184 ceph-mon[102844]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:34.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:34 smithi196 ceph-mon[76349]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:34 smithi184 ceph-mon[100622]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:34 smithi184 ceph-mon[102844]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:36.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:36 smithi196 ceph-mon[76349]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:36 smithi184 ceph-mon[100622]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:36 smithi184 ceph-mon[102844]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:38.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:38 smithi196 ceph-mon[76349]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:38 smithi184 ceph-mon[100622]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:38 smithi184 ceph-mon[102844]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:40 smithi196 ceph-mon[76349]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:40 smithi184 ceph-mon[100622]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:40 smithi184 ceph-mon[102844]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:41.705 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:00:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:41.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:41.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:42.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:41.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:00:42.231 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (5h) 17s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 17s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 17s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 73.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:00:42.232 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 17s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:00:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:00:42.542 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:00:42.543 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:00:42.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:00:42.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:00:42.544 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:00:42.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:42 smithi196 ceph-mon[76349]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:42 smithi196 ceph-mon[76349]: from='client.79022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:42.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:42 smithi184 ceph-mon[100622]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:42.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:42 smithi184 ceph-mon[100622]: from='client.79022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:42.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:42 smithi184 ceph-mon[102844]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:42.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:42 smithi184 ceph-mon[102844]: from='client.79022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:00:42.818 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:00:42.819 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:00:42.819 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:00:43.127 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:43.127 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:43.127 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:43.127 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:43.127 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[100622]: from='client.79028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[100622]: from='client.79034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/334657766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3598977713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[102844]: from='client.79028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[102844]: from='client.79034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/334657766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:43.781 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3598977713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:43.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:43 smithi196 ceph-mon[76349]: from='client.79028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:43 smithi196 ceph-mon[76349]: from='client.79034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/334657766' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3598977713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: from='client.79046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: from='client.79046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: from='client.79046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:00:46.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:46 smithi184 ceph-mon[100622]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:46.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:46 smithi184 ceph-mon[102844]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:46 smithi196 ceph-mon[76349]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:48 smithi184 ceph-mon[100622]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:48 smithi184 ceph-mon[102844]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:48 smithi196 ceph-mon[76349]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:50 smithi184 ceph-mon[100622]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:50 smithi184 ceph-mon[102844]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:50 smithi196 ceph-mon[76349]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:00:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:00:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:51.765Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:51.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:00:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:00:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:00:51.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:00:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:52 smithi184 ceph-mon[100622]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:52.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:52 smithi184 ceph-mon[102844]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:52 smithi196 ceph-mon[76349]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:54 smithi184 ceph-mon[100622]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:54 smithi184 ceph-mon[102844]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:54 smithi196 ceph-mon[76349]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:55.788 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:56.769 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:56 smithi196 ceph-mon[76349]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:56 smithi184 ceph-mon[100622]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:56.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:56 smithi184 ceph-mon[102844]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:58 smithi184 ceph-mon[100622]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:58.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:58 smithi184 ceph-mon[102844]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:58 smithi196 ceph-mon[76349]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:00 smithi184 ceph-mon[100622]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:00 smithi184 ceph-mon[102844]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:00 smithi196 ceph-mon[76349]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:01.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:01.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:01.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:01:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:02 smithi184 ceph-mon[100622]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:02 smithi184 ceph-mon[102844]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:02 smithi196 ceph-mon[76349]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:04.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[100622]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[102844]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.722 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:04 smithi196 ceph-mon[76349]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.768 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:06.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:06 smithi184 ceph-mon[100622]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:06.683 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:06 smithi184 ceph-mon[102844]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:06.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:06 smithi196 ceph-mon[76349]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:08 smithi196 ceph-mon[76349]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:08 smithi184 ceph-mon[100622]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:08 smithi184 ceph-mon[102844]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:10 smithi196 ceph-mon[76349]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:10 smithi184 ceph-mon[100622]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:10 smithi184 ceph-mon[102844]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:01:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:11.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:11.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:11.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:12 smithi196 ceph-mon[76349]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:12 smithi184 ceph-mon[100622]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:12 smithi184 ceph-mon[102844]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:13.432 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:01:13.962 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:01:13.962 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:01:13.963 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 8s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 8s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:01:13.964 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 8s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:01:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:01:14.274 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:01:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:01:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:01:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:01:14.545 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[100622]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[100622]: from='client.79058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[100622]: from='client.79064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2501617831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[102844]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[102844]: from='client.79058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[102844]: from='client.79064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2501617831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:14.857 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:01:14.857 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:14.857 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:14 smithi196 ceph-mon[76349]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:14 smithi196 ceph-mon[76349]: from='client.79058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:14 smithi196 ceph-mon[76349]: from='client.79064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2501617831' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:15.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:15 smithi196 ceph-mon[76349]: from='client.69105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:15 smithi196 ceph-mon[76349]: from='client.69117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1736507869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[100622]: from='client.69105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[100622]: from='client.69117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1736507869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[102844]: from='client.69105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[102844]: from='client.69117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1736507869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:16.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:16 smithi196 ceph-mon[76349]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:16 smithi184 ceph-mon[100622]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:16 smithi184 ceph-mon[102844]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:18 smithi196 ceph-mon[76349]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:18 smithi184 ceph-mon[100622]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:18 smithi184 ceph-mon[102844]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:20.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:20 smithi196 ceph-mon[76349]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:20 smithi184 ceph-mon[100622]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:20 smithi184 ceph-mon[102844]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:01:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:21.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:21.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:21.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:22 smithi196 ceph-mon[76349]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:22 smithi184 ceph-mon[100622]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:22 smithi184 ceph-mon[102844]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[102844]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[100622]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:24 smithi196 ceph-mon[76349]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:25.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.994 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:25.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:25.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:27.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:26 smithi184 ceph-mon[100622]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:27.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:26 smithi184 ceph-mon[102844]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:26 smithi196 ceph-mon[76349]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:28 smithi184 ceph-mon[100622]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:28 smithi184 ceph-mon[102844]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:28 smithi196 ceph-mon[76349]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:30 smithi184 ceph-mon[100622]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:30 smithi184 ceph-mon[102844]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:30 smithi196 ceph-mon[76349]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:01:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:31.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:31.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:31.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:32 smithi184 ceph-mon[102844]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:32 smithi184 ceph-mon[100622]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:33.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:32 smithi196 ceph-mon[76349]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:34 smithi184 ceph-mon[100622]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:34 smithi184 ceph-mon[102844]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:34 smithi196 ceph-mon[76349]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:36.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:36 smithi184 ceph-mon[100622]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:36 smithi184 ceph-mon[102844]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:37.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:36 smithi196 ceph-mon[76349]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:38 smithi196 ceph-mon[76349]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:38 smithi184 ceph-mon[100622]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:38 smithi184 ceph-mon[102844]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:40 smithi184 ceph-mon[102844]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:40 smithi184 ceph-mon[100622]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:40 smithi196 ceph-mon[76349]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:42.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:41 smithi184 ceph-mon[100622]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:42.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:41 smithi184 ceph-mon[102844]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:01:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:41.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:41.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:41.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:41 smithi196 ceph-mon[76349]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[100622]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[102844]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:44 smithi196 ceph-mon[76349]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:45.155 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:01:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:01:45.686 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 6h - - 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 0s ago 6h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 0s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 0s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 0s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:01:45.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (5h) 0s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 0s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 0s ago 6h 73.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:01:45.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 0s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:01:46.008 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:01:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:01:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:01:46.296 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='client.69129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='client.79100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='client.79106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4085569285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='client.69129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='client.79100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='client.79106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4085569285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:46.613 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:01:46.613 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:46.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:46.613 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:46.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='client.69129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='client.79100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='client.79106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4085569285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:47 smithi184 ceph-mon[100622]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:47.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3019531505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:47 smithi184 ceph-mon[102844]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:47.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3019531505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:47 smithi196 ceph-mon[76349]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3019531505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:48 smithi184 ceph-mon[100622]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:48 smithi184 ceph-mon[102844]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:48.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:48 smithi196 ceph-mon[76349]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:50 smithi196 ceph-mon[76349]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:50 smithi184 ceph-mon[100622]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:50 smithi184 ceph-mon[102844]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:01:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:01:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:51.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:51.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:01:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:01:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:01:51.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:01:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:52 smithi196 ceph-mon[76349]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:52 smithi184 ceph-mon[100622]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:52 smithi184 ceph-mon[102844]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:54 smithi184 ceph-mon[100622]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:54 smithi184 ceph-mon[102844]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:54 smithi196 ceph-mon[76349]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:55.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:56.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:56 smithi196 ceph-mon[76349]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:56 smithi184 ceph-mon[100622]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:56 smithi184 ceph-mon[102844]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:58 smithi196 ceph-mon[76349]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:58 smithi184 ceph-mon[100622]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:58 smithi184 ceph-mon[102844]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:00.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:00 smithi196 ceph-mon[76349]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:00 smithi184 ceph-mon[100622]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:00 smithi184 ceph-mon[102844]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:02:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:01.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:01.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:01.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:02 smithi196 ceph-mon[76349]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:02 smithi184 ceph-mon[100622]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:02 smithi184 ceph-mon[102844]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[100622]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[102844]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:04 smithi196 ceph-mon[76349]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:06 smithi196 ceph-mon[76349]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[100622]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:06.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[102844]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:06.900 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:08.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:08 smithi184 ceph-mon[100622]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:08.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:08 smithi184 ceph-mon[102844]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:08.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:08 smithi196 ceph-mon[76349]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:10 smithi196 ceph-mon[76349]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:10 smithi184 ceph-mon[100622]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:10 smithi184 ceph-mon[102844]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:02:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:11.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:11.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:11.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:12.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:12 smithi196 ceph-mon[76349]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:12 smithi184 ceph-mon[102844]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:12 smithi184 ceph-mon[100622]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:14 smithi184 ceph-mon[100622]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:14 smithi184 ceph-mon[102844]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:14.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:14 smithi196 ceph-mon[76349]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:16 smithi196 ceph-mon[76349]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:16.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:16 smithi184 ceph-mon[100622]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:16.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:16 smithi184 ceph-mon[102844]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:16.909 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:02:17.434 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 6h - - 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 13s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 13s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 13s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 13s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 13s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:02:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 13s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 13s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 13s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 13s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 11s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:02:17.436 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 11s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:02:17.743 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:02:17.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:02:17.745 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:02:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:02:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:02:18.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:02:18.322 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:02:18.322 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:18.322 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:18.323 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:18.323 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: from='client.79130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: from='client.79136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: from='client.69177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1061039506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3397268844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: from='client.79130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:19.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: from='client.79136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: from='client.69177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1061039506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:19.175 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3397268844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: from='client.79130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: from='client.79136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: from='client.69177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1061039506' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:19.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3397268844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:19.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:19 smithi196 ceph-mon[76349]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:19 smithi184 ceph-mon[100622]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:20.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:19 smithi184 ceph-mon[102844]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:20 smithi196 ceph-mon[76349]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:20 smithi184 ceph-mon[100622]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:20 smithi184 ceph-mon[102844]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:02:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:21.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:21.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:21.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:22 smithi184 ceph-mon[100622]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:22 smithi184 ceph-mon[102844]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:23.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:22 smithi196 ceph-mon[76349]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[102844]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[100622]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:25.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:24 smithi196 ceph-mon[76349]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:26.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:26.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[100622]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[102844]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:26 smithi196 ceph-mon[76349]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:29.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:28 smithi184 ceph-mon[100622]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:29.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:28 smithi184 ceph-mon[102844]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:28 smithi196 ceph-mon[76349]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:30.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:30 smithi184 ceph-mon[100622]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:30 smithi184 ceph-mon[102844]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:31.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:30 smithi196 ceph-mon[76349]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:02:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:31.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:31.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:31.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:32 smithi184 ceph-mon[100622]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:32 smithi184 ceph-mon[102844]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:32 smithi196 ceph-mon[76349]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:34 smithi184 ceph-mon[100622]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:34 smithi184 ceph-mon[102844]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:34 smithi196 ceph-mon[76349]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:36 smithi184 ceph-mon[100622]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:36 smithi184 ceph-mon[102844]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:37.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:36 smithi196 ceph-mon[76349]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:38.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:38 smithi196 ceph-mon[76349]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:38 smithi184 ceph-mon[100622]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:38 smithi184 ceph-mon[102844]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:39 smithi184 ceph-mon[100622]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:40.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:39 smithi184 ceph-mon[102844]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:40.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:39 smithi196 ceph-mon[76349]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:42.041 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:02:42.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:41.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:42.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:41.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:42.041 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:41.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:42.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:42 smithi184 ceph-mon[100622]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:42.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:42 smithi184 ceph-mon[102844]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:42 smithi196 ceph-mon[76349]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:44.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:44 smithi184 ceph-mon[102844]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:44 smithi184 ceph-mon[100622]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:44 smithi196 ceph-mon[76349]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[100622]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[102844]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:46 smithi196 ceph-mon[76349]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:02:48.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:48 smithi184 ceph-mon[102844]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:48 smithi184 ceph-mon[100622]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:48.627 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:02:48.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:48 smithi196 ceph-mon[76349]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:49.159 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:02:49.159 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 6h - - 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 3s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 3s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:02:49.160 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 3s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (5h) 3s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 3s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:02:49.161 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 3s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:02:49.481 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:02:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:02:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:02:49.482 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:02:49.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:49 smithi184 ceph-mon[102844]: from='client.69201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:49.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:49 smithi184 ceph-mon[100622]: from='client.69201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:49 smithi196 ceph-mon[76349]: from='client.69201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:02:49.769 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:02:49.770 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:02:49.770 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:02:50.090 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:02:50.090 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:50.090 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:50.090 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:50.090 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[100622]: from='client.69207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[100622]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[100622]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2414197337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4237895822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[102844]: from='client.69207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[102844]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:50.426 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[102844]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2414197337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:50.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4237895822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:50 smithi196 ceph-mon[76349]: from='client.69207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:50 smithi196 ceph-mon[76349]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:50 smithi196 ceph-mon[76349]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2414197337' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4237895822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:51.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:51 smithi184 ceph-mon[100622]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:51 smithi184 ceph-mon[102844]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:51 smithi196 ceph-mon[76349]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:02:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:02:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:51.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:51.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:02:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:02:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:02:51.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:02:52.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:52 smithi184 ceph-mon[100622]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:52.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:52 smithi184 ceph-mon[102844]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:52.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:52 smithi196 ceph-mon[76349]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:54 smithi184 ceph-mon[100622]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:54 smithi184 ceph-mon[102844]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:54.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:54 smithi196 ceph-mon[76349]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:55.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:55.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:55.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:56 smithi184 ceph-mon[100622]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:56.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:56 smithi184 ceph-mon[102844]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:56.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:56 smithi196 ceph-mon[76349]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:58.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:58 smithi196 ceph-mon[76349]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:58 smithi184 ceph-mon[100622]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:58 smithi184 ceph-mon[102844]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:59.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:00 smithi184 ceph-mon[100622]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:00 smithi184 ceph-mon[102844]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:00.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:00 smithi196 ceph-mon[76349]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:03:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:01.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:01.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:01.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:02.518 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:02 smithi184 ceph-mon[100622]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:02.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:02 smithi184 ceph-mon[102844]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:02 smithi196 ceph-mon[76349]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:04 smithi184 ceph-mon[100622]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:04 smithi184 ceph-mon[102844]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:04.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:04 smithi196 ceph-mon[76349]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:06 smithi196 ceph-mon[76349]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[102844]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[100622]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:08 smithi196 ceph-mon[76349]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:08 smithi184 ceph-mon[100622]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:08 smithi184 ceph-mon[102844]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:10.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:10 smithi196 ceph-mon[76349]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:10 smithi184 ceph-mon[100622]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:10 smithi184 ceph-mon[102844]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:03:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:11.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:11.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:11.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:12 smithi196 ceph-mon[76349]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:12.745 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:12 smithi184 ceph-mon[100622]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:12.745 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:12 smithi184 ceph-mon[102844]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:14 smithi196 ceph-mon[76349]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:14 smithi184 ceph-mon[100622]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:14 smithi184 ceph-mon[102844]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:16 smithi184 ceph-mon[100622]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:16 smithi184 ceph-mon[102844]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:16 smithi196 ceph-mon[76349]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:18.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:18 smithi196 ceph-mon[76349]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:18 smithi184 ceph-mon[100622]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:18 smithi184 ceph-mon[102844]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:20.385 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:03:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:20 smithi196 ceph-mon[76349]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:20 smithi184 ceph-mon[100622]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:20 smithi184 ceph-mon[102844]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:20.914 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 6h - - 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 16s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 16s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 16s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 16s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:03:20.915 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 16s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 16s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 16s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 16s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (5h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:03:20.916 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 14s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:03:21.227 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:03:21.227 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:03:21.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:03:21.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:03:21.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:03:21.500 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:03:21.500 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:03:21.500 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:03:21.501 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:03:21.501 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[100622]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[100622]: from='client.79208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2816135629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:21.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[102844]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[102844]: from='client.79208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.502 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2816135629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:21.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:21 smithi196 ceph-mon[76349]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:21 smithi196 ceph-mon[76349]: from='client.79208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:21.529 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2816135629' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:21.813 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:03:21.813 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:21.813 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:21.813 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:21.813 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:03:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:21.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:21.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:21.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:22.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:22 smithi196 ceph-mon[76349]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:22 smithi196 ceph-mon[76349]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:22.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:22 smithi196 ceph-mon[76349]: from='client.69261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1088675953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[102844]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[102844]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[102844]: from='client.69261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1088675953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[100622]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[100622]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[100622]: from='client.69261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1088675953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:24 smithi184 ceph-mon[100622]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:24 smithi184 ceph-mon[102844]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:24 smithi196 ceph-mon[76349]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[102844]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[100622]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:26 smithi196 ceph-mon[76349]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:28 smithi184 ceph-mon[102844]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:28 smithi184 ceph-mon[100622]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:28 smithi196 ceph-mon[76349]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:29.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:30.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:30 smithi184 ceph-mon[100622]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:30.779 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:30 smithi184 ceph-mon[102844]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:30 smithi196 ceph-mon[76349]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:31.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:31.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:31.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:31.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:31.894 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:31.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:03:32.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:32 smithi184 ceph-mon[100622]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:32.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:32 smithi184 ceph-mon[102844]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:32 smithi196 ceph-mon[76349]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:34 smithi184 ceph-mon[100622]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:34 smithi184 ceph-mon[102844]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:34 smithi196 ceph-mon[76349]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:36 smithi184 ceph-mon[100622]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:36 smithi184 ceph-mon[102844]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:36 smithi196 ceph-mon[76349]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:38 smithi184 ceph-mon[100622]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:38 smithi184 ceph-mon[102844]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:38 smithi196 ceph-mon[76349]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:40 smithi184 ceph-mon[100622]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:40 smithi184 ceph-mon[102844]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:40 smithi196 ceph-mon[76349]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:03:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:41.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:41.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:41.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:42 smithi184 ceph-mon[100622]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:42 smithi184 ceph-mon[102844]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:42 smithi196 ceph-mon[76349]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:44 smithi184 ceph-mon[100622]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:44 smithi184 ceph-mon[102844]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:44 smithi196 ceph-mon[76349]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:46 smithi196 ceph-mon[76349]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[102844]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[100622]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:03:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:48 smithi196 ceph-mon[76349]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:48 smithi184 ceph-mon[100622]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:48 smithi184 ceph-mon[102844]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:50.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:50 smithi184 ceph-mon[100622]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:50.713 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:50 smithi184 ceph-mon[102844]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:50.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:50 smithi196 ceph-mon[76349]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:52.045 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:03:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:03:52.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:51.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:52.045 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:51.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:03:52.046 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:03:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:03:51.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:03:52.111 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 6h - - 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 6s ago 6h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:03:52.675 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 6s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 6s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 6s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 6s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 6s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:03:52.676 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 6s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:03:52.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:03:52.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:03:52.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (5h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:03:52.677 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (5h) 6s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:03:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:52 smithi196 ceph-mon[76349]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:52.991 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:52 smithi184 ceph-mon[100622]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:52.991 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:52 smithi184 ceph-mon[102844]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:03:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:03:52.995 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:03:53.300 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:03:53.301 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:03:53.301 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:03:53.607 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:03:53.607 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:53.607 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:53.607 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:53.607 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[100622]: from='client.79238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[100622]: from='client.79244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[100622]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/251316445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2617505435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[102844]: from='client.79238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[102844]: from='client.79244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[102844]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/251316445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2617505435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:54.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:53 smithi196 ceph-mon[76349]: from='client.79238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:53 smithi196 ceph-mon[76349]: from='client.79244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:53 smithi196 ceph-mon[76349]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/251316445' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2617505435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:54 smithi184 ceph-mon[100622]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:54 smithi184 ceph-mon[100622]: from='client.69297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:54 smithi184 ceph-mon[102844]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:54 smithi184 ceph-mon[102844]: from='client.69297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:54 smithi196 ceph-mon[76349]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:54 smithi196 ceph-mon[76349]: from='client.69297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:56.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:56 smithi184 ceph-mon[100622]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:56 smithi184 ceph-mon[102844]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:56 smithi196 ceph-mon[76349]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:58 smithi184 ceph-mon[100622]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:58 smithi184 ceph-mon[102844]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:59.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:58 smithi196 ceph-mon[76349]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:00 smithi184 ceph-mon[100622]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:00 smithi184 ceph-mon[102844]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:00 smithi196 ceph-mon[76349]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:04:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:01.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:01.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:01.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:02 smithi184 ceph-mon[102844]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:02 smithi184 ceph-mon[100622]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:02.933 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:02 smithi196 ceph-mon[76349]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:04 smithi184 ceph-mon[100622]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:04 smithi184 ceph-mon[102844]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:04 smithi196 ceph-mon[76349]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[100622]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:04:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[102844]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:04:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:06 smithi196 ceph-mon[76349]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:04:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:08 smithi184 ceph-mon[100622]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 367 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:04:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:08 smithi184 ceph-mon[102844]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 367 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:08 smithi196 ceph-mon[76349]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 367 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:04:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:10 smithi184 ceph-mon[100622]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:04:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:10 smithi184 ceph-mon[102844]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:04:11.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:10 smithi196 ceph-mon[76349]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:04:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:04:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:11.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:11.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:11.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:12 smithi184 ceph-mon[100622]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T02:04:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:12 smithi184 ceph-mon[102844]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T02:04:13.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:12 smithi196 ceph-mon[76349]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T02:04:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:14 smithi184 ceph-mon[102844]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T02:04:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:14 smithi184 ceph-mon[100622]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T02:04:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:14 smithi196 ceph-mon[76349]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T02:04:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:16 smithi184 ceph-mon[100622]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T02:04:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:16 smithi184 ceph-mon[102844]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T02:04:17.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:16 smithi196 ceph-mon[76349]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T02:04:18.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:17 smithi196 ceph-mon[76349]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T02:04:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:17 smithi184 ceph-mon[100622]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T02:04:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:17 smithi184 ceph-mon[102844]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T02:04:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:20 smithi196 ceph-mon[76349]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:04:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:20 smithi184 ceph-mon[100622]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:04:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:20 smithi184 ceph-mon[102844]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:04:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:04:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:21.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:21.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:21.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:22.352 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:22 smithi196 ceph-mon[76349]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:04:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:22 smithi184 ceph-mon[100622]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:04:22.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:22 smithi184 ceph-mon[102844]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:04:23.904 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:04:24.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:24 smithi184 ceph-mon[100622]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:04:24.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:24 smithi184 ceph-mon[102844]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:04:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:24 smithi196 ceph-mon[76349]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 6h - - 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 19s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:04:24.439 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 19s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 19s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 19s ago 6h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 19s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 19s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 19s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 19s ago 6h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 19s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 6h 90.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:04:24.440 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:04:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:04:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:04:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:04:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:04:24.441 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:04:24.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:04:24.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:24.754 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:04:24.754 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:04:24.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:04:24.754 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:04:25.038 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:04:25.038 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:04:25.039 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:04:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='client.79280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/398899620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='client.79280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/398899620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.361 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:04:25.361 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:25.361 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:25.361 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:25.362 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='client.79280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/398899620' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:26.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:26 smithi196 ceph-mon[76349]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:04:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:26 smithi196 ceph-mon[76349]: from='client.69333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3411024988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[100622]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[100622]: from='client.69333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3411024988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[102844]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[102844]: from='client.69333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:26.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3411024988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:28 smithi184 ceph-mon[100622]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:04:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:28 smithi184 ceph-mon[102844]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:04:28.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:28 smithi196 ceph-mon[76349]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:04:29.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:30 smithi184 ceph-mon[100622]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:04:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:30 smithi184 ceph-mon[102844]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:04:30.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:30 smithi196 ceph-mon[76349]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:04:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:04:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:31.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:31.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:31.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:32 smithi184 ceph-mon[100622]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:04:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:32 smithi184 ceph-mon[102844]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:04:32.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:32 smithi196 ceph-mon[76349]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:04:34.692 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:34 smithi184 ceph-mon[100622]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:04:34.693 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:34 smithi184 ceph-mon[102844]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:04:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:34 smithi196 ceph-mon[76349]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:04:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:35.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:36 smithi184 ceph-mon[100622]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:04:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:36 smithi184 ceph-mon[102844]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:04:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:36 smithi196 ceph-mon[76349]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:04:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:38 smithi184 ceph-mon[100622]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:04:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:38 smithi184 ceph-mon[102844]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:04:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:38 smithi196 ceph-mon[76349]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:04:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:40 smithi184 ceph-mon[100622]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T02:04:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:40 smithi184 ceph-mon[102844]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T02:04:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:40 smithi196 ceph-mon[76349]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T02:04:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:04:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:41.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:41.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:41.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:42 smithi184 ceph-mon[100622]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:04:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:42 smithi184 ceph-mon[102844]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:04:42.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:42 smithi196 ceph-mon[76349]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:04:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:44 smithi196 ceph-mon[76349]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:04:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:45.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:44 smithi184 ceph-mon[100622]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:04:45.012 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:45.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:44 smithi184 ceph-mon[102844]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:04:45.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:45.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:45.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:45.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:45.857 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:46 smithi196 ceph-mon[76349]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:04:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:46 smithi184 ceph-mon[100622]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:04:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:46 smithi184 ceph-mon[102844]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:04:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:04:48.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:48 smithi196 ceph-mon[76349]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:04:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:48 smithi184 ceph-mon[100622]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:04:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:48 smithi184 ceph-mon[102844]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:04:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:50 smithi184 ceph-mon[100622]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:04:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:50 smithi184 ceph-mon[102844]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:04:51.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:50 smithi196 ceph-mon[76349]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:04:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:04:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:04:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:51.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:51.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:04:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:04:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:04:51.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:04:52.974 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:52 smithi184 ceph-mon[100622]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:04:52.974 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:52 smithi184 ceph-mon[102844]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:04:53.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:52 smithi196 ceph-mon[76349]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:04:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:54 smithi184 ceph-mon[100622]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:54 smithi184 ceph-mon[102844]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:54 smithi196 ceph-mon[76349]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:55.672 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:04:55.718 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:55.719 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:56.229 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 6h - - 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 6h - - 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 11s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 9s ago 6h 93.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 9s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 9s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 11s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:04:56.230 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 9s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 6h 98.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 6h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 9s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 9s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 9s ago 6h 97.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 9s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:04:56.231 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:04:56.553 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:04:56.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:04:56.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:04:56.555 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:04:56.833 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:04:56.834 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:04:56.834 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[100622]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[100622]: from='client.69345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2558262720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[102844]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[102844]: from='client.69345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:56.834 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2558262720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:56 smithi196 ceph-mon[76349]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:56 smithi196 ceph-mon[76349]: from='client.69345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2558262720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:57.151 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:04:57.151 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:57.152 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[100622]: from='client.79316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[100622]: from='client.79322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3158658023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[102844]: from='client.79316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[102844]: from='client.79322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3158658023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:57 smithi196 ceph-mon[76349]: from='client.79316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:57 smithi196 ceph-mon[76349]: from='client.79322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3158658023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:58 smithi184 ceph-mon[102844]: from='client.79334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:58 smithi184 ceph-mon[102844]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:58 smithi184 ceph-mon[100622]: from='client.79334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:58 smithi184 ceph-mon[100622]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:58 smithi196 ceph-mon[76349]: from='client.79334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:58 smithi196 ceph-mon[76349]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:00 smithi184 ceph-mon[100622]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:00 smithi184 ceph-mon[102844]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:01.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:00 smithi196 ceph-mon[76349]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:05:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:01.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:01.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:01.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:02 smithi184 ceph-mon[100622]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:02 smithi184 ceph-mon[102844]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:02 smithi196 ceph-mon[76349]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:04.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:04 smithi196 ceph-mon[76349]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:04 smithi184 ceph-mon[100622]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:04 smithi184 ceph-mon[102844]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:05.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:05.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:05.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:05.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:07.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:06 smithi184 ceph-mon[100622]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:07.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:06 smithi184 ceph-mon[102844]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:06 smithi196 ceph-mon[76349]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:08 smithi184 ceph-mon[100622]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:09.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:08 smithi184 ceph-mon[102844]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:08 smithi196 ceph-mon[76349]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:10 smithi184 ceph-mon[100622]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:10 smithi184 ceph-mon[102844]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:10 smithi196 ceph-mon[76349]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:05:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:11.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:11.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:11.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:12 smithi184 ceph-mon[100622]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:12 smithi184 ceph-mon[102844]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:12 smithi196 ceph-mon[76349]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:14.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:14 smithi184 ceph-mon[100622]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:14.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:14.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:14 smithi184 ceph-mon[102844]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:14.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:14 smithi196 ceph-mon[76349]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:16.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:16 smithi184 ceph-mon[102844]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:16 smithi184 ceph-mon[100622]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:16 smithi196 ceph-mon[76349]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:18.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:17 smithi184 ceph-mon[102844]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:17 smithi184 ceph-mon[100622]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:18.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:17 smithi196 ceph-mon[76349]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:20 smithi196 ceph-mon[76349]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:20 smithi184 ceph-mon[100622]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:20 smithi184 ceph-mon[102844]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:05:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:21.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:21.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:21.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:22.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:22 smithi196 ceph-mon[76349]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:22 smithi184 ceph-mon[100622]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:22.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:22 smithi184 ceph-mon[102844]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:24.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:24 smithi196 ceph-mon[76349]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:24 smithi184 ceph-mon[100622]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:24 smithi184 ceph-mon[102844]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:25.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:26.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:26 smithi196 ceph-mon[76349]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:26.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.510 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.511 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[100622]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[102844]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:27.446 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:05:27.975 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:05:27.975 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T02:05:27.975 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 2s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 2s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 2s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 2s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 2s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 2s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 2s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 2s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 2s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 2s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:05:27.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 6h 97.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:05:27.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: from='client.79346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:27 smithi196 ceph-mon[76349]: from='client.69387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:05:28.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:05:28.285 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:05:28.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: from='client.79346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[100622]: from='client.69387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: from='client.79346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:27 smithi184 ceph-mon[102844]: from='client.69387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:05:28.559 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:05:28.560 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:05:28.560 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:05:28.868 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:05:28.869 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:28.869 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:28.869 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:28.869 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:28 smithi196 ceph-mon[76349]: from='client.69393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2110576576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:28 smithi196 ceph-mon[76349]: from='client.69405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1635524764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[100622]: from='client.69393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2110576576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[100622]: from='client.69405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1635524764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[102844]: from='client.69393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2110576576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[102844]: from='client.69405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1635524764' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:29 smithi184 ceph-mon[100622]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:29 smithi184 ceph-mon[102844]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:29 smithi196 ceph-mon[76349]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:05:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:31.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:31.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:31.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:32.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:32 smithi196 ceph-mon[76349]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:32.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:32 smithi184 ceph-mon[100622]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:32.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:32 smithi184 ceph-mon[102844]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:34 smithi196 ceph-mon[76349]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:34 smithi184 ceph-mon[100622]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.414 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:34 smithi184 ceph-mon[102844]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:35.425 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:35.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:36 smithi196 ceph-mon[76349]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:36 smithi184 ceph-mon[100622]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:36 smithi184 ceph-mon[102844]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:38 smithi196 ceph-mon[76349]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:38 smithi184 ceph-mon[100622]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:38 smithi184 ceph-mon[102844]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:40.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:40 smithi196 ceph-mon[76349]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:40 smithi184 ceph-mon[100622]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:40.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:40 smithi184 ceph-mon[102844]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:05:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:41.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:41.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:41.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:42 smithi196 ceph-mon[76349]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:42 smithi184 ceph-mon[100622]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:42.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:42 smithi184 ceph-mon[102844]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:44.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:44 smithi196 ceph-mon[76349]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:44 smithi184 ceph-mon[100622]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:44 smithi184 ceph-mon[102844]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:45.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:45.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:45.396 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:45.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:45.397 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:46 smithi196 ceph-mon[76349]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[100622]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[102844]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[100622]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:48 smithi184 ceph-mon[102844]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:05:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:48 smithi196 ceph-mon[76349]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:50.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:50 smithi196 ceph-mon[76349]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:50 smithi184 ceph-mon[100622]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:50.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:50 smithi184 ceph-mon[102844]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:05:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:05:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:51.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:51.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:05:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:05:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:05:51.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:05:52.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:52 smithi196 ceph-mon[76349]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:52.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:52 smithi184 ceph-mon[100622]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:52.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:52 smithi184 ceph-mon[102844]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:54 smithi196 ceph-mon[76349]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:54 smithi184 ceph-mon[100622]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:54.707 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:54 smithi184 ceph-mon[102844]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:56 smithi184 ceph-mon[100622]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:56 smithi184 ceph-mon[102844]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:56.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:56 smithi196 ceph-mon[76349]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:05:58 smithi196 ceph-mon[76349]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:05:58 smithi184 ceph-mon[100622]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:05:58 smithi184 ceph-mon[102844]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:59.166 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:05:59.695 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:05:59.695 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T02:05:59.695 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T02:05:59.695 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:05:59.696 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 12s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:05:59.697 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:05:59.697 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:05:59.697 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:05:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:05:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:05:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:05:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:05:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 12s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:05:59.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 12s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:05:59.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 12s ago 6h 97.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:05:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 12s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:05:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 12s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:05:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:05:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:05:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:05:59.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 12s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:06:00.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:06:00.008 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:06:00.277 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:06:00.277 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:06:00.277 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:06:00.277 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:06:00.277 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:06:00.278 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:06:00.279 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:06:00.279 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:06:00.279 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:06:00.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: from='client.69417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/891584314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:00.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:00.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:00.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: from='client.69417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.281 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/891584314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: from='client.69417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/891584314' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:00.590 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:06:00.590 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:00.590 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:01 smithi196 ceph-mon[76349]: from='client.69441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2098809061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-mon[100622]: from='client.69441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2098809061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-mon[102844]: from='client.69441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2098809061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:06:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:01.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:01.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:01.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:02 smithi196 ceph-mon[76349]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:02.508 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:02 smithi184 ceph-mon[100622]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:02.508 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:02 smithi184 ceph-mon[102844]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:04 smithi196 ceph-mon[76349]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:04 smithi184 ceph-mon[100622]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:04 smithi184 ceph-mon[102844]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:05.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:05.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:06 smithi196 ceph-mon[76349]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[100622]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[102844]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:08 smithi196 ceph-mon[76349]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:08.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[100622]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[102844]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:10 smithi184 ceph-mon[100622]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:10.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:10 smithi184 ceph-mon[102844]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:10 smithi196 ceph-mon[76349]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:06:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:11.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:11.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:11.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:12.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:12 smithi184 ceph-mon[100622]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:12.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:12 smithi184 ceph-mon[102844]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:12 smithi196 ceph-mon[76349]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:14 smithi184 ceph-mon[100622]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:14 smithi184 ceph-mon[102844]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:14.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:14 smithi196 ceph-mon[76349]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:15.524 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:15.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:16.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:16 smithi184 ceph-mon[100622]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:16.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:16 smithi184 ceph-mon[102844]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:16 smithi196 ceph-mon[76349]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:18 smithi184 ceph-mon[100622]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:18 smithi184 ceph-mon[102844]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:18.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:18 smithi196 ceph-mon[76349]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:20 smithi184 ceph-mon[100622]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:20 smithi184 ceph-mon[102844]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:20 smithi196 ceph-mon[76349]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:06:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:21.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:21.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:21.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:22 smithi184 ceph-mon[100622]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:22.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:22 smithi184 ceph-mon[102844]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:22 smithi196 ceph-mon[76349]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:24 smithi184 ceph-mon[100622]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:24 smithi184 ceph-mon[102844]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:24.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:24 smithi196 ceph-mon[76349]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:25.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:26 smithi196 ceph-mon[76349]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[102844]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[100622]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[100622]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[102844]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:28 smithi196 ceph-mon[76349]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:29.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:30 smithi196 ceph-mon[76349]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:30.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:30 smithi184 ceph-mon[100622]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:30.890 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:30 smithi184 ceph-mon[102844]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:30.890 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:06:31.416 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:06:31.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:06:31.418 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 6h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:06:31.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:06:31.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:06:31.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:06:31.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:06:31.726 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:06:31.995 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:06:31.996 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:06:31.996 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:06:31.996 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:06:31.996 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:06:31.996 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:06:31.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:31.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:31.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:31.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:31.996 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:31.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:06:32.306 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:06:32.306 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:32.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:32.307 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:32.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: from='client.69453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: from='client.79424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2661254061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2712653195' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: from='client.69453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: from='client.79424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2661254061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2712653195' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: from='client.69453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: from='client.79424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2661254061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2712653195' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:33.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:33 smithi184 ceph-mon[100622]: from='client.69477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:33 smithi184 ceph-mon[102844]: from='client.69477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:33.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:33 smithi196 ceph-mon[76349]: from='client.69477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:34.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:34 smithi196 ceph-mon[76349]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:34 smithi184 ceph-mon[100622]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:34 smithi184 ceph-mon[102844]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:36.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:36 smithi184 ceph-mon[100622]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:36 smithi184 ceph-mon[102844]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:36 smithi196 ceph-mon[76349]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:38 smithi196 ceph-mon[76349]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:38 smithi184 ceph-mon[100622]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:38 smithi184 ceph-mon[102844]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:40 smithi196 ceph-mon[76349]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:40 smithi184 ceph-mon[100622]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:40 smithi184 ceph-mon[102844]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:06:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:41.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:41.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:42.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:41.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:42.982 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:42 smithi184 ceph-mon[100622]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:42.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:42 smithi184 ceph-mon[102844]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:42 smithi196 ceph-mon[76349]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:44 smithi184 ceph-mon[100622]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:44 smithi184 ceph-mon[102844]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:45.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:44 smithi196 ceph-mon[76349]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[100622]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[102844]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:46 smithi196 ceph-mon[76349]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:48 smithi196 ceph-mon[76349]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:48.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[100622]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[102844]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:06:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:50 smithi184 ceph-mon[100622]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:50 smithi184 ceph-mon[102844]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:50 smithi196 ceph-mon[76349]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:52.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:06:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:06:52.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:51.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:51.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:06:52.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:06:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:06:51.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:06:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:52 smithi184 ceph-mon[100622]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:52 smithi184 ceph-mon[102844]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:52.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:52 smithi196 ceph-mon[76349]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:54.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:54 smithi184 ceph-mon[100622]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:54.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:54 smithi184 ceph-mon[102844]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:54.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:54 smithi196 ceph-mon[76349]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:55.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:56 smithi184 ceph-mon[100622]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:56.828 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:56 smithi184 ceph-mon[102844]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:56 smithi196 ceph-mon[76349]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:58 smithi184 ceph-mon[100622]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:06:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:58 smithi184 ceph-mon[102844]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:06:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:58 smithi196 ceph-mon[76349]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:06:59.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:59.750 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:00 smithi184 ceph-mon[100622]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:00 smithi184 ceph-mon[102844]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:00 smithi196 ceph-mon[76349]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:07:02.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:01.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:01.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:01.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:02.609 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:07:02.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:02 smithi184 ceph-mon[100622]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:02.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:02 smithi184 ceph-mon[102844]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:02 smithi196 ceph-mon[76349]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:03.139 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 6h - - 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 17s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 17s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 17s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 17s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 15s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 17s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 17s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 17s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:07:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 17s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 17s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 17s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 15s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 15s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 15s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 15s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 15s ago 6h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:07:03.141 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 15s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:07:03.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:07:03.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:07:03.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:07:03.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:07:03.449 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:07:03.720 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:07:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:07:03.721 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:07:03.721 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:07:03.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:03 smithi184 ceph-mon[100622]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:03.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/397949175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:03.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:03 smithi184 ceph-mon[102844]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:03.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/397949175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:03 smithi196 ceph-mon[76349]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/397949175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:07:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:04.031 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:04.032 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[100622]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[100622]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[100622]: from='client.79466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[100622]: from='client.69513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1837856925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[102844]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.716 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[102844]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:04.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[102844]: from='client.79466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[102844]: from='client.69513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1837856925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:04 smithi196 ceph-mon[76349]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:04 smithi196 ceph-mon[76349]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:04 smithi196 ceph-mon[76349]: from='client.79466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:04 smithi196 ceph-mon[76349]: from='client.69513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1837856925' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:05.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:05.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[100622]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[102844]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:06 smithi196 ceph-mon[76349]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:08 smithi196 ceph-mon[76349]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[102844]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[100622]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:10.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:10 smithi196 ceph-mon[76349]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:10 smithi184 ceph-mon[100622]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:10 smithi184 ceph-mon[102844]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:07:12.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:11.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:11.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:11.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:12 smithi196 ceph-mon[76349]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:12 smithi184 ceph-mon[100622]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:12 smithi184 ceph-mon[102844]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:14 smithi196 ceph-mon[76349]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:14 smithi184 ceph-mon[100622]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:14 smithi184 ceph-mon[102844]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:16 smithi196 ceph-mon[76349]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:16 smithi184 ceph-mon[100622]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:16 smithi184 ceph-mon[102844]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:18.863 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:18 smithi184 ceph-mon[100622]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:07:18.864 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:18 smithi184 ceph-mon[102844]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:07:18.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:18 smithi196 ceph-mon[76349]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:07:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:20 smithi196 ceph-mon[76349]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:20 smithi184 ceph-mon[100622]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:20 smithi184 ceph-mon[102844]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:07:22.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:21.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:21.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:21.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:22 smithi196 ceph-mon[76349]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:22 smithi184 ceph-mon[100622]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:22 smithi184 ceph-mon[102844]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:24.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:24 smithi196 ceph-mon[76349]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:24 smithi184 ceph-mon[100622]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:24 smithi184 ceph-mon[102844]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:26.037 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:26.037 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:26 smithi196 ceph-mon[76349]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[102844]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[100622]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:28.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:28 smithi196 ceph-mon[76349]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:28.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:28.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:28.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[100622]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[102844]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:29.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:30 smithi196 ceph-mon[76349]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:30 smithi184 ceph-mon[100622]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:30 smithi184 ceph-mon[102844]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:07:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:31.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:32.047 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:31.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:31.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:32 smithi184 ceph-mon[100622]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:32 smithi184 ceph-mon[102844]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:32 smithi196 ceph-mon[76349]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:34.327 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 6h - - 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 7s ago 6h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 7s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:07:34.855 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 7s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 7s ago 6h 74.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:07:34.856 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:07:34.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:07:34.857 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:07:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:34 smithi184 ceph-mon[100622]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:34 smithi184 ceph-mon[102844]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:35.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:34 smithi196 ceph-mon[76349]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:07:35.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:07:35.168 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:07:35.437 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:07:35.438 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:07:35.745 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:07:35.745 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:35.745 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:35.745 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:35.745 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[100622]: from='client.69525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[100622]: from='client.69531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:36.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3056956024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[102844]: from='client.69525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[102844]: from='client.69531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3056956024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:35 smithi196 ceph-mon[76349]: from='client.69525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:35 smithi196 ceph-mon[76349]: from='client.69531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3056956024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[100622]: from='client.69537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[100622]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[100622]: from='client.69549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2943845394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[102844]: from='client.69537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[102844]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[102844]: from='client.69549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2943845394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:36 smithi196 ceph-mon[76349]: from='client.69537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:36 smithi196 ceph-mon[76349]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:36 smithi196 ceph-mon[76349]: from='client.69549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2943845394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:38 smithi184 ceph-mon[100622]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:38 smithi184 ceph-mon[102844]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:38 smithi196 ceph-mon[76349]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:40 smithi184 ceph-mon[100622]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:40 smithi184 ceph-mon[102844]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:40 smithi196 ceph-mon[76349]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:07:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:41.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:41.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:41.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:42 smithi184 ceph-mon[100622]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:42 smithi184 ceph-mon[102844]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:42 smithi196 ceph-mon[76349]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:44 smithi184 ceph-mon[100622]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:44 smithi184 ceph-mon[102844]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:44 smithi196 ceph-mon[76349]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:46.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[100622]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[102844]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:46 smithi196 ceph-mon[76349]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:48 smithi196 ceph-mon[76349]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[100622]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[102844]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:07:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:50 smithi196 ceph-mon[76349]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:50 smithi184 ceph-mon[100622]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:50 smithi184 ceph-mon[102844]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:07:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:07:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:51.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:51.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:07:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:07:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:07:51.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:07:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:52 smithi196 ceph-mon[76349]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:52 smithi184 ceph-mon[100622]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:52 smithi184 ceph-mon[102844]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:54 smithi184 ceph-mon[100622]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:54 smithi184 ceph-mon[102844]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:54 smithi196 ceph-mon[76349]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:55 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:55 smithi196 ceph-mon[76349]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:55 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:56.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:55 smithi184 ceph-mon[100622]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:56.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:55 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:55 smithi184 ceph-mon[102844]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:58 smithi184 ceph-mon[100622]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:58.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:58 smithi184 ceph-mon[102844]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:58 smithi196 ceph-mon[76349]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:59.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:59.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:00 smithi184 ceph-mon[100622]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:00 smithi184 ceph-mon[102844]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:00 smithi196 ceph-mon[76349]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:02.161 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:08:02.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:01.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:02.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:01.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:02.161 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:01.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:02.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:02 smithi184 ceph-mon[100622]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:02.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:02 smithi184 ceph-mon[102844]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:02 smithi196 ceph-mon[76349]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:04 smithi184 ceph-mon[100622]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:04 smithi184 ceph-mon[102844]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:04 smithi196 ceph-mon[76349]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:05.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:05 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:05.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:05 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:05.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:05 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:06.041 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:08:06.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:06 smithi184 ceph-mon[100622]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:06 smithi184 ceph-mon[102844]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.616 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:08:06.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 6h - - 2024-09-21T02:08:06.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 0s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 92.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 0s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 0s ago 6h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 0s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 0s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 0s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:08:06.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 6h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 6h 74.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:08:06.618 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:08:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:06 smithi196 ceph-mon[76349]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:08:06.927 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:08:06.928 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:08:07.194 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:08:07.194 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:08:07.195 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:08:07.501 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:08:07.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:07.502 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:07.502 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:07.502 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='client.69561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.502 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2782348702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='client.69561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2782348702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='client.69561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2782348702' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: from='client.69585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1279975210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: from='client.69585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1279975210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: from='client.69585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1279975210' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:10.621 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:10 smithi196 ceph-mon[76349]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:10 smithi184 ceph-mon[100622]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:10 smithi184 ceph-mon[102844]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:08:12.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:11.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:11.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:11.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:12.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:12 smithi196 ceph-mon[76349]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:12 smithi184 ceph-mon[100622]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:12 smithi184 ceph-mon[102844]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:08:13.812540364Z level=info msg="Completed cleanup jobs" duration=12.579164ms 2024-09-21T02:08:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:14 smithi196 ceph-mon[76349]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:14.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:14 smithi184 ceph-mon[100622]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.784 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:14.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:14 smithi184 ceph-mon[102844]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.784 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:15.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:15 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:15.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:15 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:15.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:15 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:16.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:16 smithi196 ceph-mon[76349]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:16 smithi184 ceph-mon[100622]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:16 smithi184 ceph-mon[102844]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:18 smithi196 ceph-mon[76349]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:18 smithi184 ceph-mon[100622]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:18 smithi184 ceph-mon[102844]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:20 smithi196 ceph-mon[76349]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:20.683 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:20 smithi184 ceph-mon[100622]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:20.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:20 smithi184 ceph-mon[102844]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:21.825 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:21.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:21.825 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:21.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:21.826 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:21.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:08:22.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:22 smithi196 ceph-mon[76349]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:22 smithi184 ceph-mon[100622]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:22 smithi184 ceph-mon[102844]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:24 smithi196 ceph-mon[76349]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:24.795 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:24 smithi184 ceph-mon[100622]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:24.795 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:24 smithi184 ceph-mon[102844]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:25 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:25 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:25 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:26 smithi196 ceph-mon[76349]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:26 smithi184 ceph-mon[100622]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:26 smithi184 ceph-mon[102844]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[100622]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[102844]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:28 smithi196 ceph-mon[76349]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:28.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:30 smithi184 ceph-mon[100622]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:30 smithi184 ceph-mon[102844]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:30 smithi196 ceph-mon[76349]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:08:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:31.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:31.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:31.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:32 smithi196 ceph-mon[76349]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:32.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:32 smithi184 ceph-mon[100622]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:32 smithi184 ceph-mon[102844]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:34 smithi184 ceph-mon[100622]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:34 smithi184 ceph-mon[102844]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:34 smithi196 ceph-mon[76349]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:35.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:35 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:35.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:35 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:35.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:35 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:36 smithi184 ceph-mon[100622]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:36 smithi184 ceph-mon[102844]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:36 smithi196 ceph-mon[76349]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:37.796 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:08:38.321 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:08:38.321 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 6h - - 2024-09-21T02:08:38.321 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 6h - - 2024-09-21T02:08:38.321 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 11s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 10s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 10s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 10s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 11s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 10s ago 6h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:08:38.322 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 6h 74.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:08:38.323 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:08:38.633 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:08:38.633 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:08:38.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:08:38.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:08:38.635 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:08:38.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:38 smithi184 ceph-mon[102844]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:38.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:38 smithi184 ceph-mon[102844]: from='client.69597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:38.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:38 smithi184 ceph-mon[100622]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:38.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:38 smithi184 ceph-mon[100622]: from='client.69597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:38 smithi196 ceph-mon[76349]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:38 smithi196 ceph-mon[76349]: from='client.69597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:38.901 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:08:38.902 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:08:38.903 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:08:39.209 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:08:39.209 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:39.209 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:39.209 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:39.209 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[100622]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[100622]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1997706491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:39.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1309836701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[102844]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[102844]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1997706491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1309836701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:39 smithi196 ceph-mon[76349]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:39 smithi196 ceph-mon[76349]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1997706491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1309836701' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:40 smithi184 ceph-mon[102844]: from='client.69621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:40 smithi184 ceph-mon[102844]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:40 smithi184 ceph-mon[100622]: from='client.69621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:40 smithi184 ceph-mon[100622]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:40 smithi196 ceph-mon[76349]: from='client.69621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:40 smithi196 ceph-mon[76349]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:42.181 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:41.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:42.181 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:41.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:42.181 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:41.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:42.181 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:08:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:42 smithi184 ceph-mon[100622]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:42 smithi184 ceph-mon[102844]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:42 smithi196 ceph-mon[76349]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:44 smithi184 ceph-mon[100622]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:44 smithi184 ceph-mon[102844]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:44 smithi196 ceph-mon[76349]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:45.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:45 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:45.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:45 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:45 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:46 smithi184 ceph-mon[100622]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:46 smithi184 ceph-mon[102844]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:46 smithi196 ceph-mon[76349]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:48 smithi196 ceph-mon[76349]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[100622]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[102844]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:08:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:50 smithi196 ceph-mon[76349]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:50 smithi184 ceph-mon[100622]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:50 smithi184 ceph-mon[102844]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:08:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:08:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:51.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:51.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:08:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:08:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:08:51.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:08:52.843 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:52 smithi196 ceph-mon[76349]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:52 smithi184 ceph-mon[100622]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:52 smithi184 ceph-mon[102844]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:54 smithi196 ceph-mon[76349]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:54 smithi184 ceph-mon[100622]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:54 smithi184 ceph-mon[102844]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:56 smithi196 ceph-mon[76349]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:56 smithi184 ceph-mon[100622]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:56 smithi184 ceph-mon[102844]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:58 smithi196 ceph-mon[76349]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:58 smithi184 ceph-mon[100622]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:58 smithi184 ceph-mon[102844]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:00 smithi196 ceph-mon[76349]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:00.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:00 smithi184 ceph-mon[100622]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:00.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:00 smithi184 ceph-mon[102844]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:09:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:01.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:01.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:01.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:02 smithi196 ceph-mon[76349]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:02 smithi184 ceph-mon[100622]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:02 smithi184 ceph-mon[102844]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:04 smithi196 ceph-mon[76349]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:04 smithi184 ceph-mon[100622]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:04 smithi184 ceph-mon[102844]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:06 smithi196 ceph-mon[76349]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:06 smithi184 ceph-mon[100622]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:06 smithi184 ceph-mon[102844]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:08.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[100622]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:09:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[102844]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:09:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:08 smithi196 ceph-mon[76349]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:09:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:09.507 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 6h - - 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 3s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:09:10.037 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 3s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 3s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 3s ago 6h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 3s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 3s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 3s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 3s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:09:10.038 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:09:10.039 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 75.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:09:10.039 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:09:10.039 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:09:10.039 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:09:10.039 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:09:10.345 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:09:10.345 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:09:10.346 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:09:10.614 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:09:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:09:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:09:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:09:10.615 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:09:10.615 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:09:10.922 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:09:10.922 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:10.922 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:10.923 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[100622]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[100622]: from='client.69633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[100622]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2009820306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[102844]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:10.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[102844]: from='client.69633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:10.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[102844]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:10.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2009820306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:10 smithi196 ceph-mon[76349]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:10 smithi196 ceph-mon[76349]: from='client.69633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:10 smithi196 ceph-mon[76349]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2009820306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[100622]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[100622]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2600136245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[102844]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[102844]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2600136245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:11.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:11.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:12.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:11.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:12.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:11 smithi196 ceph-mon[76349]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:11 smithi196 ceph-mon[76349]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:12.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2600136245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:09:13.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:12 smithi184 ceph-mon[100622]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:13.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:12 smithi184 ceph-mon[102844]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:13.128 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:12 smithi196 ceph-mon[76349]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:14 smithi196 ceph-mon[76349]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:14 smithi184 ceph-mon[100622]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:14 smithi184 ceph-mon[102844]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:16 smithi184 ceph-mon[100622]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:16 smithi184 ceph-mon[102844]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:16 smithi196 ceph-mon[76349]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:18 smithi184 ceph-mon[100622]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:18 smithi184 ceph-mon[102844]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:18 smithi196 ceph-mon[76349]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:20 smithi184 ceph-mon[100622]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:20 smithi184 ceph-mon[102844]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:20 smithi196 ceph-mon[76349]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:22.224 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:09:22.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:21.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:22.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:21.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:22.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:21.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:23.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:22 smithi184 ceph-mon[100622]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:23.033 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:22 smithi184 ceph-mon[102844]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:22 smithi196 ceph-mon[76349]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:24 smithi184 ceph-mon[100622]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:24 smithi184 ceph-mon[102844]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:24 smithi196 ceph-mon[76349]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:26 smithi184 ceph-mon[100622]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:26 smithi184 ceph-mon[102844]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:26 smithi196 ceph-mon[76349]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:27 smithi196 ceph-mon[76349]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[100622]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:27 smithi184 ceph-mon[102844]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:30 smithi184 ceph-mon[100622]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:30.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:30 smithi184 ceph-mon[102844]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:30 smithi196 ceph-mon[76349]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:09:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:31.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:31.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:31.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:32.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:32 smithi196 ceph-mon[76349]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:32.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:32 smithi184 ceph-mon[100622]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:32.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:32 smithi184 ceph-mon[102844]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:34.697 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:34 smithi196 ceph-mon[76349]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:34 smithi184 ceph-mon[100622]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:34 smithi184 ceph-mon[102844]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:36 smithi184 ceph-mon[102844]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:36 smithi184 ceph-mon[100622]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:36 smithi196 ceph-mon[76349]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:38 smithi184 ceph-mon[100622]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:38 smithi184 ceph-mon[102844]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:38 smithi196 ceph-mon[76349]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:40 smithi184 ceph-mon[100622]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:40 smithi184 ceph-mon[102844]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:40 smithi196 ceph-mon[76349]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:41.223 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:09:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:09:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 6h - - 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 13s ago 6h 91.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 13s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 13s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:09:41.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 13s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 13s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 13s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 13s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 6h 75.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:09:41.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 13s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:09:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:41.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:41.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:41.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:42.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:09:42.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:09:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:09:42.348 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:09:42.349 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:09:42.667 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[100622]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:42.667 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[100622]: from='client.69669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.667 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[100622]: from='client.69675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.667 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[100622]: from='client.69681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4008451009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[102844]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[102844]: from='client.69669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[102844]: from='client.69675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[102844]: from='client.69681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.668 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4008451009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:42.668 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:09:42.668 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:42.669 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:42 smithi196 ceph-mon[76349]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:42 smithi196 ceph-mon[76349]: from='client.69669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:42 smithi196 ceph-mon[76349]: from='client.69675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:42 smithi196 ceph-mon[76349]: from='client.69681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4008451009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:43 smithi184 ceph-mon[100622]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:43.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3860919457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:43 smithi184 ceph-mon[102844]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:43.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3860919457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:43 smithi196 ceph-mon[76349]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3860919457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:44 smithi184 ceph-mon[100622]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:44 smithi184 ceph-mon[102844]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:44.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:44 smithi196 ceph-mon[76349]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:46 smithi184 ceph-mon[100622]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:46 smithi184 ceph-mon[102844]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:46 smithi196 ceph-mon[76349]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:48.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:47 smithi196 ceph-mon[76349]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[100622]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:48.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:48.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:47 smithi184 ceph-mon[102844]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:09:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:50 smithi196 ceph-mon[76349]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:50 smithi184 ceph-mon[100622]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:50 smithi184 ceph-mon[102844]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:09:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:09:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:51.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:51.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:09:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:09:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:09:51.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:09:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:52 smithi196 ceph-mon[76349]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:52 smithi184 ceph-mon[100622]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:52 smithi184 ceph-mon[102844]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:54.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:54 smithi196 ceph-mon[76349]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:54.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:54 smithi184 ceph-mon[102844]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:54.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:54 smithi184 ceph-mon[100622]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:56 smithi196 ceph-mon[76349]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:56 smithi184 ceph-mon[102844]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:56 smithi184 ceph-mon[100622]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:58 smithi196 ceph-mon[76349]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:58 smithi184 ceph-mon[100622]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:58 smithi184 ceph-mon[102844]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:00.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:00.000 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:09:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:00 smithi196 ceph-mon[76349]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:00 smithi184 ceph-mon[100622]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:00 smithi184 ceph-mon[102844]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:10:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:01.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:01.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:01.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:02.860 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:02 smithi184 ceph-mon[100622]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:02 smithi184 ceph-mon[102844]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:02.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:02 smithi196 ceph-mon[76349]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:04.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:04 smithi184 ceph-mon[100622]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:04.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:04 smithi184 ceph-mon[102844]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:04.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:04 smithi196 ceph-mon[76349]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:06 smithi196 ceph-mon[76349]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:06 smithi184 ceph-mon[100622]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:06 smithi184 ceph-mon[102844]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:08 smithi196 ceph-mon[76349]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[100622]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:08 smithi184 ceph-mon[102844]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:10:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:10 smithi184 ceph-mon[100622]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:11.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:10 smithi184 ceph-mon[102844]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:10 smithi196 ceph-mon[76349]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:11.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:11.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:11.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:12.969 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:10:13.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:12 smithi184 ceph-mon[100622]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:13.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:12 smithi184 ceph-mon[102844]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:12 smithi196 ceph-mon[76349]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:13.495 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:10:13.495 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 6h - - 2024-09-21T02:10:13.495 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T02:10:13.495 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 6s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 6s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 6s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 6s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 6s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 6s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 6s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 6s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 6s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:10:13.496 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 6s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 6h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:10:13.497 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:10:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:10:13.817 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:10:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:10:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:10:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:10:14.091 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:10:14.410 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:14.410 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:14.410 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:14.410 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='client.79670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='client.79676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/239944628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/736089672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='client.79670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='client.79676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/239944628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/736089672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='client.79670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='client.79676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/239944628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:15.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/736089672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:15.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:15 smithi196 ceph-mon[76349]: from='client.69729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:15 smithi184 ceph-mon[100622]: from='client.69729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:16.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:15 smithi184 ceph-mon[102844]: from='client.69729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:16 smithi184 ceph-mon[100622]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:16 smithi184 ceph-mon[102844]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:16 smithi196 ceph-mon[76349]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:18 smithi184 ceph-mon[100622]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:18 smithi184 ceph-mon[102844]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:18 smithi196 ceph-mon[76349]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:20 smithi184 ceph-mon[100622]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:20 smithi184 ceph-mon[102844]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:20 smithi196 ceph-mon[76349]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:10:22.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:21.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:21.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:21.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:22 smithi184 ceph-mon[102844]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:22 smithi184 ceph-mon[100622]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:22 smithi196 ceph-mon[76349]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:25.016 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:24 smithi184 ceph-mon[100622]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:25.307 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:24 smithi184 ceph-mon[102844]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:25.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:24 smithi196 ceph-mon[76349]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:26 smithi184 ceph-mon[100622]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:26 smithi184 ceph-mon[102844]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:26 smithi196 ceph-mon[76349]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[100622]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[102844]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:28 smithi196 ceph-mon[76349]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:29 smithi196 ceph-mon[76349]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[100622]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:29 smithi184 ceph-mon[102844]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:32.100 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:10:32.101 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:31.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:32.101 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:31.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:32.101 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:31.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:32 smithi184 ceph-mon[100622]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:32 smithi184 ceph-mon[102844]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:32 smithi196 ceph-mon[76349]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:34 smithi184 ceph-mon[100622]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:34.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:34 smithi184 ceph-mon[102844]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:34 smithi196 ceph-mon[76349]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:36 smithi196 ceph-mon[76349]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:36.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:36 smithi184 ceph-mon[100622]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:36 smithi184 ceph-mon[102844]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:38 smithi184 ceph-mon[100622]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:38 smithi184 ceph-mon[102844]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:38 smithi196 ceph-mon[76349]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:40 smithi184 ceph-mon[100622]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:40 smithi184 ceph-mon[102844]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:40 smithi196 ceph-mon[76349]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:42.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:10:42.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:41.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:42.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:41.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:42.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:41.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:42.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:42 smithi184 ceph-mon[102844]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:42.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:42 smithi184 ceph-mon[100622]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:42 smithi196 ceph-mon[76349]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:44 smithi184 ceph-mon[100622]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:44 smithi184 ceph-mon[102844]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:44 smithi196 ceph-mon[76349]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:44.708 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 6h - - 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 16s ago 6h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:10:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 16s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 16s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 16s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 16s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:10:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 16s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 16s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 16s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 16s ago 6h 75.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:10:45.269 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 16s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:10:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:45 smithi184 ceph-mon[100622]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:45 smithi184 ceph-mon[102844]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:10:45.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:10:45.590 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:10:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:45 smithi196 ceph-mon[76349]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:10:45.903 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:10:45.904 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:10:45.904 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:10:45.904 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:10:46.230 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:46.230 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:46.230 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:46.230 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:46.230 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: from='client.69747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: from='client.79718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2302569452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4131247934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: from='client.69747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: from='client.79718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2302569452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4131247934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: from='client.69747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: from='client.79718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2302569452' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4131247934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:47 smithi184 ceph-mon[100622]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:47.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:47 smithi184 ceph-mon[102844]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:47 smithi196 ceph-mon[76349]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:48 smithi196 ceph-mon[76349]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[100622]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[102844]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[100622]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:49 smithi184 ceph-mon[102844]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:10:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:49 smithi196 ceph-mon[76349]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:52.217 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:10:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:10:52.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:51.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:52.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:51.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:10:52.218 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:10:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:10:51.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:10:52.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:52 smithi184 ceph-mon[100622]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:52.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:52 smithi184 ceph-mon[102844]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:52 smithi196 ceph-mon[76349]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:54 smithi184 ceph-mon[100622]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:54.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:54 smithi184 ceph-mon[102844]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:54 smithi196 ceph-mon[76349]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:56.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:56.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:56 smithi196 ceph-mon[76349]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:56.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:56 smithi184 ceph-mon[100622]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:56.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:56.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:56 smithi184 ceph-mon[102844]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:58.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:58 smithi184 ceph-mon[100622]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:58 smithi184 ceph-mon[102844]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:58 smithi196 ceph-mon[76349]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:10:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:10:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:10:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:00 smithi184 ceph-mon[100622]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:00 smithi184 ceph-mon[102844]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:00.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:00 smithi196 ceph-mon[76349]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:02.239 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:11:02.239 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:01.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:02.239 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:01.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:02.239 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:01.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:02.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:02 smithi184 ceph-mon[100622]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:02.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:02 smithi184 ceph-mon[102844]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:02 smithi196 ceph-mon[76349]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:04 smithi184 ceph-mon[100622]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:04 smithi184 ceph-mon[102844]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:04 smithi196 ceph-mon[76349]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:06 smithi184 ceph-mon[100622]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:06 smithi184 ceph-mon[102844]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:06 smithi196 ceph-mon[76349]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[100622]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[102844]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:08 smithi196 ceph-mon[76349]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:10 smithi196 ceph-mon[76349]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[100622]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:10 smithi184 ceph-mon[102844]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:12.224 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:11:12.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:11.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:12.225 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:11.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:12.225 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:11.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:12.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:12 smithi184 ceph-mon[100622]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:12.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:12 smithi184 ceph-mon[102844]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:12.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:12 smithi196 ceph-mon[76349]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:14 smithi184 ceph-mon[100622]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:14.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:14 smithi184 ceph-mon[102844]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:14 smithi196 ceph-mon[76349]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:16 smithi196 ceph-mon[76349]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:16.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:16.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:16 smithi184 ceph-mon[100622]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:16.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:16.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:16 smithi184 ceph-mon[102844]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:16.527 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:11:17.053 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 6h - - 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 9s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 7s ago 6h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 9s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 7s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 9s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 9s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 9s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 7s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:11:17.054 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 9s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 9s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 9s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 9s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 7s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:11:17.055 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:17.367 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:11:17.368 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:11:17.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:17 smithi184 ceph-mon[100622]: from='client.69777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.368 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:17 smithi184 ceph-mon[100622]: from='client.79748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.368 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:17 smithi184 ceph-mon[102844]: from='client.69777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:17 smithi184 ceph-mon[102844]: from='client.79748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:17 smithi196 ceph-mon[76349]: from='client.69777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:17 smithi196 ceph-mon[76349]: from='client.79748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:17.637 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:11:17.638 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:11:17.639 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:11:17.949 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:11:17.949 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:17.949 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:17.950 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:17.950 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[100622]: from='client.79754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[100622]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1523133980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[100622]: from='client.69801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1549452356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[102844]: from='client.79754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[102844]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1523133980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[102844]: from='client.69801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1549452356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:18 smithi196 ceph-mon[76349]: from='client.79754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:18 smithi196 ceph-mon[76349]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1523133980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:18 smithi196 ceph-mon[76349]: from='client.69801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1549452356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:20 smithi184 ceph-mon[100622]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:20.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:20 smithi184 ceph-mon[102844]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:20 smithi196 ceph-mon[76349]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:22.244 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:11:22.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:21.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:22.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:21.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:22.245 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:21.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:22.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:22 smithi184 ceph-mon[102844]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:22.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:22 smithi184 ceph-mon[100622]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:22 smithi196 ceph-mon[76349]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:24.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:24 smithi184 ceph-mon[102844]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:24.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:24 smithi184 ceph-mon[100622]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:24 smithi196 ceph-mon[76349]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:26.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:26.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:26 smithi184 ceph-mon[102844]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:26.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:26.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:26 smithi184 ceph-mon[100622]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:26 smithi196 ceph-mon[76349]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:28 smithi196 ceph-mon[76349]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[100622]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[102844]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:30 smithi196 ceph-mon[76349]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:30 smithi184 ceph-mon[100622]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:30 smithi184 ceph-mon[102844]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:11:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:31.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:31.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:31.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:32 smithi196 ceph-mon[76349]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:32 smithi184 ceph-mon[100622]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:32 smithi184 ceph-mon[102844]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:34 smithi196 ceph-mon[76349]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:34.959 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:34 smithi184 ceph-mon[100622]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:34.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:34 smithi184 ceph-mon[102844]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:36.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:36 smithi196 ceph-mon[76349]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:36 smithi184 ceph-mon[100622]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:36 smithi184 ceph-mon[102844]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:38.869 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:38 smithi196 ceph-mon[76349]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:38 smithi184 ceph-mon[100622]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:38 smithi184 ceph-mon[102844]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:40.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:40 smithi196 ceph-mon[76349]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:40 smithi184 ceph-mon[100622]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:40 smithi184 ceph-mon[102844]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:11:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:41.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:41.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:41.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:42 smithi196 ceph-mon[76349]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:43.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:42 smithi184 ceph-mon[100622]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:43.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:42 smithi184 ceph-mon[102844]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:44 smithi196 ceph-mon[76349]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:44.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:44 smithi184 ceph-mon[100622]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:44.970 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:44.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:44 smithi184 ceph-mon[102844]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:11:44.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:46.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:46 smithi184 ceph-mon[102844]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:46 smithi184 ceph-mon[100622]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:46 smithi196 ceph-mon[76349]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:48.241 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:11:48.780 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:11:48.780 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T02:11:48.780 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 6h - - 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 19s ago 6h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 19s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 19s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 19s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:11:48.781 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 19s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 19s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 19s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 19s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 19s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:11:48.782 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 19s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:11:49.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[100622]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:11:49.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[102844]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:11:49.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.112 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:11:49.113 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:11:49.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:11:49.114 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:11:49.114 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:48 smithi196 ceph-mon[76349]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:11:49.405 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:11:49.406 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:11:49.406 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:11:49.406 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:11:49.723 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:11:49.724 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:49.724 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:49.724 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:49.724 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:49.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='client.69813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='client.79784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3194565122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='client.69813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='client.79784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3194565122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:49.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='client.69813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='client.79784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3194565122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:11:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[100622]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[100622]: from='client.69837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3521443982' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[102844]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[102844]: from='client.69837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3521443982' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:50 smithi196 ceph-mon[76349]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:50 smithi196 ceph-mon[76349]: from='client.69837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3521443982' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:11:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:11:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:51.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:51.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:11:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:11:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:11:51.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:11:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:52 smithi184 ceph-mon[100622]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:52 smithi184 ceph-mon[102844]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:52 smithi196 ceph-mon[76349]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:54 smithi184 ceph-mon[100622]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:54 smithi184 ceph-mon[102844]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:55.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:54 smithi196 ceph-mon[76349]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:56 smithi184 ceph-mon[100622]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:56 smithi184 ceph-mon[102844]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:56 smithi196 ceph-mon[76349]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:58.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:58 smithi196 ceph-mon[76349]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:58 smithi184 ceph-mon[100622]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:58 smithi184 ceph-mon[102844]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:00.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:00 smithi184 ceph-mon[102844]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:00 smithi184 ceph-mon[100622]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:00 smithi196 ceph-mon[76349]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:12:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:01.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:01.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:01.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:02 smithi184 ceph-mon[100622]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:02 smithi184 ceph-mon[102844]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:03.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:02 smithi196 ceph-mon[76349]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:04 smithi184 ceph-mon[100622]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:04 smithi184 ceph-mon[102844]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:04 smithi196 ceph-mon[76349]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:06.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:06.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:06 smithi184 ceph-mon[100622]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:06.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:06.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:06 smithi184 ceph-mon[102844]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:06 smithi196 ceph-mon[76349]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[100622]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[102844]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:08 smithi196 ceph-mon[76349]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[100622]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[102844]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:10 smithi196 ceph-mon[76349]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:12:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:11.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:11.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:11.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:12 smithi184 ceph-mon[100622]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:12.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:12 smithi184 ceph-mon[102844]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:12 smithi196 ceph-mon[76349]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:14 smithi184 ceph-mon[100622]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:14 smithi184 ceph-mon[102844]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:14 smithi196 ceph-mon[76349]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:16 smithi184 ceph-mon[102844]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:16 smithi184 ceph-mon[100622]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:16 smithi196 ceph-mon[76349]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:18 smithi196 ceph-mon[76349]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:18 smithi184 ceph-mon[100622]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:18 smithi184 ceph-mon[102844]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:20.020 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 12s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 12s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:12:20.552 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 12s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 12s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 12s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 71.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:12:20.553 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:12:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:20 smithi184 ceph-mon[100622]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:20 smithi184 ceph-mon[102844]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:12:20.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:12:20.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:20.864 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:12:20.864 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:12:20.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:12:20.864 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:12:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:20 smithi196 ceph-mon[76349]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:21.132 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:12:21.132 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:12:21.132 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:12:21.133 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:12:21.443 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:12:21.443 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:21.443 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:21.443 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:21.443 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[100622]: from='client.69849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[100622]: from='client.79820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[100622]: from='client.69861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2679952793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3412362033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[102844]: from='client.69849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[102844]: from='client.79820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[102844]: from='client.69861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2679952793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3412362033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:21 smithi196 ceph-mon[76349]: from='client.69849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:21 smithi196 ceph-mon[76349]: from='client.79820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:21 smithi196 ceph-mon[76349]: from='client.69861 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2679952793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3412362033' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:22] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:12:22.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:21.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:21.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:21.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:22 smithi184 ceph-mon[100622]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:22 smithi184 ceph-mon[100622]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:22 smithi184 ceph-mon[102844]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:22 smithi184 ceph-mon[102844]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:22 smithi196 ceph-mon[76349]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:22 smithi196 ceph-mon[76349]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:24 smithi184 ceph-mon[100622]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:24 smithi184 ceph-mon[102844]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:24 smithi196 ceph-mon[76349]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:26 smithi184 ceph-mon[100622]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:26 smithi184 ceph-mon[102844]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:26 smithi196 ceph-mon[76349]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:28.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[100622]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[102844]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.657 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:28 smithi196 ceph-mon[76349]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:30 smithi196 ceph-mon[76349]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[100622]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[102844]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:12:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:31.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:32.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:31.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:31.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:32 smithi196 ceph-mon[76349]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:32 smithi184 ceph-mon[102844]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:32 smithi184 ceph-mon[100622]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:34 smithi196 ceph-mon[76349]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:35.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:34 smithi184 ceph-mon[100622]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:35.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:34 smithi184 ceph-mon[102844]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:36 smithi196 ceph-mon[76349]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:36 smithi184 ceph-mon[100622]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:36 smithi184 ceph-mon[102844]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:38 smithi196 ceph-mon[76349]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:38 smithi184 ceph-mon[100622]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:38 smithi184 ceph-mon[102844]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:40 smithi196 ceph-mon[76349]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:40 smithi184 ceph-mon[100622]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:40 smithi184 ceph-mon[102844]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:12:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:41.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:41.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:41.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:42 smithi184 ceph-mon[100622]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:42 smithi184 ceph-mon[102844]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:42 smithi196 ceph-mon[76349]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:45.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:44 smithi184 ceph-mon[100622]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:45.006 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:45.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:44 smithi184 ceph-mon[102844]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:45.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:44 smithi196 ceph-mon[76349]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:46 smithi184 ceph-mon[100622]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:46 smithi184 ceph-mon[102844]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:46 smithi196 ceph-mon[76349]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[100622]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[102844]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:48.952 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:48 smithi196 ceph-mon[76349]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[100622]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[102844]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:50 smithi196 ceph-mon[76349]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:51.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:12:51.738 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:12:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:51.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:51.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:12:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:12:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:12:51.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 6h - - 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 2s ago 6h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 2s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 6h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:12:52.269 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 2s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 2s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 2s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 2s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 2s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 2s ago 6h 72.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:12:52.270 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 2s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:12:52.271 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:12:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:12:52.583 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:12:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:12:52.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:12:52.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:12:52.855 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:12:52.856 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[100622]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[100622]: from='client.79850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3777939646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[102844]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[102844]: from='client.79850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3777939646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:52 smithi196 ceph-mon[76349]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:52 smithi196 ceph-mon[76349]: from='client.79850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3777939646' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:12:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:53.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:53.163 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:53.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[102844]: from='client.69891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[102844]: from='client.69897 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[102844]: from='client.69909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/416703758' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[100622]: from='client.69891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[100622]: from='client.69897 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[100622]: from='client.69909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/416703758' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:53 smithi196 ceph-mon[76349]: from='client.69891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:53 smithi196 ceph-mon[76349]: from='client.69897 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:53 smithi196 ceph-mon[76349]: from='client.69909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/416703758' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:55.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:54 smithi184 ceph-mon[100622]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:55.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:54 smithi184 ceph-mon[102844]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:54 smithi196 ceph-mon[76349]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:56 smithi184 ceph-mon[102844]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:56 smithi184 ceph-mon[100622]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:56 smithi196 ceph-mon[76349]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:58 smithi184 ceph-mon[100622]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:58 smithi184 ceph-mon[102844]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:59.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:58 smithi196 ceph-mon[76349]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:00.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:01.039 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:00 smithi196 ceph-mon[76349]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:00 smithi184 ceph-mon[100622]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:00 smithi184 ceph-mon[102844]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:13:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:01.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:01.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:01.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:02 smithi184 ceph-mon[100622]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:02 smithi184 ceph-mon[102844]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:02 smithi196 ceph-mon[76349]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:04 smithi184 ceph-mon[100622]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:04 smithi184 ceph-mon[102844]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:04 smithi196 ceph-mon[76349]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:06 smithi184 ceph-mon[100622]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:06 smithi184 ceph-mon[102844]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:06 smithi196 ceph-mon[76349]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:08.939 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[100622]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[102844]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:08.940 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:08 smithi196 ceph-mon[76349]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[100622]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[102844]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:10.889 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:11.095 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:10 smithi196 ceph-mon[76349]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:11.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:11.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:11.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:13:12.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:11.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:11.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:11.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:12 smithi184 ceph-mon[100622]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:12 smithi184 ceph-mon[102844]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:12 smithi196 ceph-mon[76349]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:14 smithi184 ceph-mon[100622]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:14 smithi184 ceph-mon[102844]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:15.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:14 smithi196 ceph-mon[76349]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:16 smithi196 ceph-mon[76349]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:16 smithi184 ceph-mon[100622]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:16 smithi184 ceph-mon[102844]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:18 smithi196 ceph-mon[76349]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:18 smithi184 ceph-mon[100622]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:18 smithi184 ceph-mon[102844]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:21.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:20 smithi196 ceph-mon[76349]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:20 smithi184 ceph-mon[100622]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:21.321 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:20 smithi184 ceph-mon[102844]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:13:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:21.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:21.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:21.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:22 smithi196 ceph-mon[76349]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:22 smithi184 ceph-mon[100622]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:22 smithi184 ceph-mon[102844]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.461 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:13:23.983 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 6h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:13:23.984 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 72.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:13:23.985 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:13:24.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:13:24.299 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:13:24.568 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:13:24.568 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:13:24.569 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:13:24.884 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:13:24.884 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:24.885 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[100622]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[100622]: from='client.79886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[100622]: from='client.79892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1378024068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[102844]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:24.885 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[102844]: from='client.79886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:24.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[102844]: from='client.79892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:24.886 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1378024068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:24 smithi196 ceph-mon[76349]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:24 smithi196 ceph-mon[76349]: from='client.79886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:24 smithi196 ceph-mon[76349]: from='client.79892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1378024068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:25 smithi196 ceph-mon[76349]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:25 smithi196 ceph-mon[76349]: from='client.69945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4004936671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:26.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[102844]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[102844]: from='client.69945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4004936671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[100622]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[100622]: from='client.69945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4004936671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:26 smithi196 ceph-mon[76349]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:26 smithi184 ceph-mon[102844]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:26 smithi184 ceph-mon[100622]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:28 smithi196 ceph-mon[76349]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[100622]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[102844]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:29.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:30.271 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:30.271 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[100622]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[102844]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:30.928 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:30 smithi196 ceph-mon[76349]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:31.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:31.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:31.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:31.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:31.908 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:31.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:13:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:32 smithi196 ceph-mon[76349]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:32 smithi184 ceph-mon[100622]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:32 smithi184 ceph-mon[102844]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:34 smithi196 ceph-mon[76349]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:34 smithi184 ceph-mon[100622]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:34 smithi184 ceph-mon[102844]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:36 smithi196 ceph-mon[76349]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:36 smithi184 ceph-mon[100622]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:36 smithi184 ceph-mon[102844]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:38 smithi196 ceph-mon[76349]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:38 smithi184 ceph-mon[100622]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:38 smithi184 ceph-mon[102844]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:40.973 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:40 smithi196 ceph-mon[76349]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:40 smithi184 ceph-mon[100622]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:41.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:40 smithi184 ceph-mon[102844]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:13:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:41.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:41.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:41.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:42 smithi196 ceph-mon[76349]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:42 smithi184 ceph-mon[100622]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:43.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:42 smithi184 ceph-mon[102844]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:44 smithi196 ceph-mon[76349]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:44 smithi184 ceph-mon[100622]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:44 smithi184 ceph-mon[102844]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:46 smithi196 ceph-mon[76349]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:46 smithi184 ceph-mon[100622]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:46 smithi184 ceph-mon[102844]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:48 smithi196 ceph-mon[76349]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[100622]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[102844]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[100622]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:51.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[102844]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:50 smithi196 ceph-mon[76349]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:13:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:51.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:51.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:13:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:13:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:13:51.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:13:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:13:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:13:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:52 smithi196 ceph-mon[76349]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:53.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:52 smithi184 ceph-mon[100622]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:53.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:52 smithi184 ceph-mon[102844]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:54 smithi196 ceph-mon[76349]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:55.176 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:54 smithi184 ceph-mon[100622]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:55.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:54 smithi184 ceph-mon[102844]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:55.177 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:13:55.707 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:13:55.707 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 5s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:13:55.708 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 5s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 5s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 5s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 5s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:13:55.709 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 5s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:13:56.022 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:13:56.022 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:13:56.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:13:56.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:13:56.024 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:13:56.293 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:13:56.294 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:13:56.294 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:13:56.294 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:13:56.294 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:13:56.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:13:56.603 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:13:56.603 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:56.603 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:56.603 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:56.603 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.79928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.69969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/488304170' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2898820970' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.79928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.69969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/488304170' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2898820970' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.79928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.69969 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/488304170' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2898820970' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:57 smithi196 ceph-mon[76349]: from='client.79946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:57 smithi196 ceph-mon[76349]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:57 smithi184 ceph-mon[100622]: from='client.79946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:57 smithi184 ceph-mon[100622]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:57 smithi184 ceph-mon[102844]: from='client.79946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:57 smithi184 ceph-mon[102844]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:59.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:59.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:00.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:00 smithi184 ceph-mon[102844]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:00 smithi184 ceph-mon[100622]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:00 smithi196 ceph-mon[76349]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:02.287 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:14:02.287 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:01.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:02.287 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:01.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:02.287 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:01.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:02.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:02 smithi184 ceph-mon[100622]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:02.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:02 smithi184 ceph-mon[102844]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:02.619 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:02 smithi196 ceph-mon[76349]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:04 smithi196 ceph-mon[76349]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:04.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:04 smithi184 ceph-mon[100622]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:04.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:04 smithi184 ceph-mon[102844]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:06.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:06 smithi184 ceph-mon[100622]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:06 smithi184 ceph-mon[102844]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:06.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:06 smithi196 ceph-mon[76349]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:08.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:08 smithi184 ceph-mon[100622]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:14:08.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:08 smithi184 ceph-mon[102844]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:14:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:08 smithi196 ceph-mon[76349]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:14:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[102844]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[100622]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:10.866 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:10 smithi196 ceph-mon[76349]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:14:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:11.961 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:11.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:11.961 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:11.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:11.961 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:11.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:12.250 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:14:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:12 smithi184 ceph-mon[100622]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:13.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:12 smithi184 ceph-mon[102844]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:12 smithi196 ceph-mon[76349]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:14 smithi184 ceph-mon[100622]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:14 smithi184 ceph-mon[102844]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:14 smithi196 ceph-mon[76349]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:14:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:16 smithi184 ceph-mon[100622]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:14:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:16 smithi184 ceph-mon[102844]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:14:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:16 smithi196 ceph-mon[76349]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:14:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:18 smithi184 ceph-mon[100622]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T02:14:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:18 smithi184 ceph-mon[102844]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T02:14:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:18 smithi196 ceph-mon[76349]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T02:14:20.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:20 smithi196 ceph-mon[76349]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T02:14:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:20 smithi184 ceph-mon[100622]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T02:14:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:20 smithi184 ceph-mon[102844]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 58 op/s 2024-09-21T02:14:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:21.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:21.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:21.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:22.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:22 smithi196 ceph-mon[76349]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:14:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:22 smithi184 ceph-mon[100622]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:14:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:22 smithi184 ceph-mon[102844]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:14:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:24 smithi184 ceph-mon[100622]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:14:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:24 smithi184 ceph-mon[102844]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:14:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:24 smithi196 ceph-mon[76349]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:14:26.900 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:26 smithi184 ceph-mon[100622]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:14:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:26 smithi184 ceph-mon[102844]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:26 smithi196 ceph-mon[76349]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 392 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:14:27.426 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:14:27.426 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T02:14:27.426 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T02:14:27.426 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:14:27.426 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 6h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 6h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:14:27.427 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 6h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 6h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 72.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:14:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:14:27.733 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:14:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:14:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:14:27.735 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:14:28.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:14:28.003 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:14:28.315 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:14:28.315 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:28.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:28.316 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:28.316 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: from='client.79958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: from='client.69999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:14:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: from='client.70005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2168179930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1016048020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: from='client.79958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: from='client.69999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: from='client.70005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2168179930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1016048020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: from='client.79958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: from='client.69999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: from='client.70005 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2168179930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1016048020' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[100622]: from='client.70017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[102844]: from='client.70017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:29 smithi196 ceph-mon[76349]: from='client.70017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[100622]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:14:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[102844]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:14:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:30 smithi196 ceph-mon[76349]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 379 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:32.292 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:14:32.292 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:31.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:32.292 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:31.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:32.292 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:31.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:32.878 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:32 smithi184 ceph-mon[100622]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:14:32.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:32 smithi184 ceph-mon[102844]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:14:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:32 smithi196 ceph-mon[76349]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T02:14:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:34 smithi184 ceph-mon[100622]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T02:14:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:34 smithi184 ceph-mon[102844]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T02:14:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:34 smithi196 ceph-mon[76349]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T02:14:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:36 smithi184 ceph-mon[100622]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:36 smithi184 ceph-mon[102844]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:36 smithi196 ceph-mon[76349]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:38 smithi184 ceph-mon[100622]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:38 smithi184 ceph-mon[102844]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:39.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:38 smithi196 ceph-mon[76349]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:14:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:40 smithi184 ceph-mon[100622]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T02:14:41.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:40 smithi184 ceph-mon[102844]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T02:14:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:40 smithi196 ceph-mon[76349]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T02:14:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:41.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:41.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:41.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:42 smithi184 ceph-mon[100622]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:14:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:42 smithi184 ceph-mon[102844]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:14:43.058 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:42 smithi196 ceph-mon[76349]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:14:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:44 smithi184 ceph-mon[100622]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:14:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:44 smithi184 ceph-mon[102844]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:14:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:45.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:44 smithi196 ceph-mon[76349]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:14:45.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:46 smithi184 ceph-mon[100622]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:14:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:46 smithi184 ceph-mon[102844]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:14:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:46 smithi196 ceph-mon[76349]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:14:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:48 smithi184 ceph-mon[100622]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:14:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:48 smithi184 ceph-mon[102844]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:14:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:48 smithi196 ceph-mon[76349]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:14:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:49 smithi196 ceph-mon[76349]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:14:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[100622]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:49 smithi184 ceph-mon[102844]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T02:14:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:14:52.147 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:14:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:14:52.147 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:51.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:52.147 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:51.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:14:52.147 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:14:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:14:51.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:14:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:52 smithi184 ceph-mon[100622]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:14:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:52 smithi184 ceph-mon[102844]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:14:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:52 smithi196 ceph-mon[76349]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:14:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:54 smithi196 ceph-mon[76349]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:54 smithi184 ceph-mon[100622]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:54 smithi184 ceph-mon[102844]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:56 smithi196 ceph-mon[76349]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:56 smithi184 ceph-mon[100622]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:56 smithi184 ceph-mon[102844]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:58.615 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:14:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:58 smithi196 ceph-mon[76349]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:58 smithi184 ceph-mon[100622]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:58 smithi184 ceph-mon[102844]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:14:59.142 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 8s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 8s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 8s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:14:59.143 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 8s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:14:59.456 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:14:59.456 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:14:59.456 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:14:59.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:14:59.457 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:14:59.728 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:14:59.729 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:14:59.729 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:14:59.729 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:14:59.729 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:14:59.729 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:14:59.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[100622]: from='client.79994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[100622]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:59.729 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2361463859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:59.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[102844]: from='client.79994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[102844]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:59.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:14:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2361463859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:59 smithi196 ceph-mon[76349]: from='client.79994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:59 smithi196 ceph-mon[76349]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:14:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2361463859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:00.040 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:15:00.040 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:00.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:00.040 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:00.040 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:00 smithi196 ceph-mon[76349]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:00 smithi196 ceph-mon[76349]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:00 smithi196 ceph-mon[76349]: from='client.70053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4062821476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[100622]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[100622]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[100622]: from='client.70053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4062821476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[102844]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[102844]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[102844]: from='client.70053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4062821476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:01.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:01.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:01.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:02 smithi196 ceph-mon[76349]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:02 smithi184 ceph-mon[100622]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:02 smithi184 ceph-mon[102844]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:04.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:04 smithi196 ceph-mon[76349]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:04 smithi184 ceph-mon[100622]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:04 smithi184 ceph-mon[102844]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:06 smithi196 ceph-mon[76349]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:06 smithi184 ceph-mon[100622]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:06 smithi184 ceph-mon[102844]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:08 smithi196 ceph-mon[76349]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:09.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:08 smithi184 ceph-mon[100622]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:09.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:08 smithi184 ceph-mon[102844]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[100622]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:10 smithi184 ceph-mon[102844]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:10 smithi196 ceph-mon[76349]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:11.829Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:12.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:11.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:12.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:11.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:15:12.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:12 smithi184 ceph-mon[100622]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:12.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:12 smithi184 ceph-mon[102844]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:12 smithi196 ceph-mon[76349]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:14 smithi184 ceph-mon[100622]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:14 smithi184 ceph-mon[102844]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:15.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:14 smithi196 ceph-mon[76349]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:16 smithi196 ceph-mon[76349]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:16 smithi184 ceph-mon[102844]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:16 smithi184 ceph-mon[100622]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:18 smithi184 ceph-mon[100622]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:18 smithi184 ceph-mon[102844]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:18 smithi196 ceph-mon[76349]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:20.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:20 smithi196 ceph-mon[76349]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:20 smithi184 ceph-mon[100622]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:20 smithi184 ceph-mon[102844]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:21.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:21.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:21.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:22 smithi184 ceph-mon[100622]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:22 smithi184 ceph-mon[102844]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:22 smithi196 ceph-mon[76349]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:25.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:24 smithi196 ceph-mon[76349]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:24 smithi184 ceph-mon[100622]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:25.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:24 smithi184 ceph-mon[102844]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:26 smithi184 ceph-mon[100622]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:26 smithi184 ceph-mon[102844]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:26 smithi196 ceph-mon[76349]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:28 smithi184 ceph-mon[100622]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:29.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:28 smithi184 ceph-mon[102844]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:29.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:28 smithi196 ceph-mon[76349]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:30.344 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:15:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:30 smithi196 ceph-mon[76349]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[102844]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:30 smithi184 ceph-mon[100622]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 6h - - 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 0s ago 6h 96.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 0s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:15:30.879 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 0s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 0s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 0s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 0s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 0s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 0s ago 6h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:15:30.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:15:30.881 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:15:31.191 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:15:31.192 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:15:31.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:15:31.193 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:15:31.193 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:15:31.461 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:15:31.462 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:15:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:15:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:31.771 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1020729375' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1626432990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:32.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:15:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:31.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:31.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:31.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1020729375' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:32.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1626432990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1020729375' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:32.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1626432990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:32 smithi196 ceph-mon[76349]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:32 smithi196 ceph-mon[76349]: from='client.80054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:33.229 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:32 smithi184 ceph-mon[100622]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:33.229 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:32 smithi184 ceph-mon[100622]: from='client.80054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:33.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:32 smithi184 ceph-mon[102844]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:33.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:32 smithi184 ceph-mon[102844]: from='client.80054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:34 smithi184 ceph-mon[100622]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:34 smithi184 ceph-mon[102844]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:34 smithi196 ceph-mon[76349]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:36 smithi196 ceph-mon[76349]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:36 smithi184 ceph-mon[100622]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:36 smithi184 ceph-mon[102844]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:38 smithi196 ceph-mon[76349]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:38 smithi184 ceph-mon[102844]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:38 smithi184 ceph-mon[100622]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:40 smithi196 ceph-mon[76349]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:40 smithi184 ceph-mon[100622]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:40 smithi184 ceph-mon[102844]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:15:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:41.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:41.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:41.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:42 smithi184 ceph-mon[100622]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:42 smithi184 ceph-mon[102844]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:43.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:42 smithi196 ceph-mon[76349]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:44 smithi196 ceph-mon[76349]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:45.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:44 smithi184 ceph-mon[100622]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:45.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:45.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:44 smithi184 ceph-mon[102844]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:45.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:47.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:47.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:46 smithi196 ceph-mon[76349]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:46 smithi184 ceph-mon[100622]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:46 smithi184 ceph-mon[102844]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:48 smithi196 ceph-mon[76349]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:49.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:48 smithi184 ceph-mon[100622]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:49.288 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:48 smithi184 ceph-mon[102844]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[100622]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[102844]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:50 smithi196 ceph-mon[76349]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:15:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[100622]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-mon[102844]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:51.834Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:51.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:15:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:15:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:15:51.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:15:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:15:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:51 smithi196 ceph-mon[76349]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:54 smithi184 ceph-mon[100622]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:54 smithi184 ceph-mon[102844]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:54 smithi196 ceph-mon[76349]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:56 smithi184 ceph-mon[100622]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:56 smithi184 ceph-mon[102844]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:56 smithi196 ceph-mon[76349]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:58 smithi196 ceph-mon[76349]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:58 smithi184 ceph-mon[100622]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:58 smithi184 ceph-mon[102844]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:00 smithi196 ceph-mon[76349]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:00 smithi184 ceph-mon[100622]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:00 smithi184 ceph-mon[102844]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:02.067 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:16:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:16:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:01.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:01.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:01.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:02.594 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 6h - - 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 13s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 96.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 13s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 13s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 13s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 13s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:16:02.595 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 13s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 13s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 13s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 13s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:16:02.596 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:16:02.597 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:16:02.597 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:02 smithi184 ceph-mon[100622]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:02.597 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:02 smithi184 ceph-mon[102844]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:02 smithi196 ceph-mon[76349]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:16:02.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:16:02.901 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:16:03.169 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:16:03.169 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:16:03.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:16:03.483 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:16:03.483 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:03.483 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:03.483 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:03.483 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:03.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[102844]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.483 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[102844]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[102844]: from='client.80078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1699905059' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[100622]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[100622]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[100622]: from='client.80078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1699905059' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:03 smithi196 ceph-mon[76349]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:03 smithi196 ceph-mon[76349]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:03 smithi196 ceph-mon[76349]: from='client.80078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1699905059' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:04 smithi196 ceph-mon[76349]: from='client.70125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:04 smithi196 ceph-mon[76349]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2657576399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[102844]: from='client.70125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[102844]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2657576399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[100622]: from='client.70125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[100622]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2657576399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:06.464 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:06.464 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:06 smithi196 ceph-mon[76349]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:06 smithi184 ceph-mon[100622]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:06 smithi184 ceph-mon[102844]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:08 smithi196 ceph-mon[76349]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:08 smithi184 ceph-mon[100622]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:08 smithi184 ceph-mon[102844]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[100622]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[102844]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:10 smithi196 ceph-mon[76349]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[100622]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:12 smithi184 ceph-mon[102844]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:11.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:11.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:11.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:12 smithi196 ceph-mon[76349]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:14.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:14 smithi184 ceph-mon[100622]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:14.633 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:14.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:14 smithi184 ceph-mon[102844]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:14.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:14 smithi196 ceph-mon[76349]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:16 smithi196 ceph-mon[76349]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:16 smithi184 ceph-mon[100622]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:16 smithi184 ceph-mon[102844]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:18 smithi196 ceph-mon[76349]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:18 smithi184 ceph-mon[100622]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:18.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:18 smithi184 ceph-mon[102844]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:20 smithi196 ceph-mon[76349]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:20 smithi184 ceph-mon[102844]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:20 smithi184 ceph-mon[100622]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:16:22.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:21.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:21.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:21.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:22 smithi196 ceph-mon[76349]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:22 smithi184 ceph-mon[100622]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:22.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:22 smithi184 ceph-mon[102844]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:24 smithi196 ceph-mon[76349]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:24 smithi184 ceph-mon[100622]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:24.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:24 smithi184 ceph-mon[102844]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:26 smithi196 ceph-mon[76349]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:26 smithi184 ceph-mon[100622]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:26 smithi184 ceph-mon[102844]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:28 smithi196 ceph-mon[76349]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:28 smithi184 ceph-mon[100622]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:28 smithi184 ceph-mon[102844]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[100622]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[102844]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:30 smithi196 ceph-mon[76349]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.186 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:16:32.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:31.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:32.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:31.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:32.186 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:31.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[100622]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:32 smithi184 ceph-mon[102844]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:32.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:32 smithi196 ceph-mon[76349]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:33.789 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 6h - - 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:16:34.331 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 3s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 3s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 3s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 3s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:16:34.332 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 3s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:16:34.333 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 3s ago 6h 71.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:16:34.333 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:16:34.333 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:16:34.333 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:16:34.333 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 3s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:16:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:34 smithi196 ceph-mon[76349]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:34 smithi196 ceph-mon[76349]: from='client.80102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:16:34.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:16:34.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:16:34.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:34 smithi184 ceph-mon[100622]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:34.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:34 smithi184 ceph-mon[100622]: from='client.80102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:34.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:34 smithi184 ceph-mon[102844]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:34.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:34 smithi184 ceph-mon[102844]: from='client.80102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:34.913 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:16:34.913 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:16:34.913 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:16:34.913 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:16:34.913 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:16:34.914 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:16:35.230 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:16:35.231 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:35.231 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:35.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:35.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[100622]: from='client.80108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[100622]: from='client.80114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3564253356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:35.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2483189999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:35.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[102844]: from='client.80108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[102844]: from='client.80114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3564253356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:35.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2483189999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:35 smithi196 ceph-mon[76349]: from='client.80108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:35 smithi196 ceph-mon[76349]: from='client.80114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3564253356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2483189999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:36.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[100622]: from='client.80126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:36.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:36.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[100622]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[102844]: from='client.80126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:36.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:36 smithi184 ceph-mon[102844]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:36 smithi196 ceph-mon[76349]: from='client.80126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:36 smithi196 ceph-mon[76349]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:38 smithi196 ceph-mon[76349]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:38 smithi184 ceph-mon[100622]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:38 smithi184 ceph-mon[102844]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:40 smithi196 ceph-mon[76349]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:40 smithi184 ceph-mon[100622]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:40 smithi184 ceph-mon[102844]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:16:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:41.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:41.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:41.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:42 smithi196 ceph-mon[76349]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:42 smithi184 ceph-mon[100622]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:42.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:42 smithi184 ceph-mon[102844]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:44 smithi196 ceph-mon[76349]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:44 smithi184 ceph-mon[100622]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:44 smithi184 ceph-mon[102844]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:46.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:46.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:46 smithi196 ceph-mon[76349]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:46 smithi184 ceph-mon[100622]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:46 smithi184 ceph-mon[102844]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:48 smithi196 ceph-mon[76349]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:48.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:48 smithi184 ceph-mon[100622]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:48.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:48 smithi184 ceph-mon[102844]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:50.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:50 smithi196 ceph-mon[76349]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:50.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:50.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:50.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[100622]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[102844]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:16:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:16:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:51.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:51.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:16:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:16:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:16:51.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:52 smithi196 ceph-mon[76349]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[100622]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[102844]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:16:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:54 smithi196 ceph-mon[76349]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:54 smithi184 ceph-mon[100622]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:54.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:54 smithi184 ceph-mon[102844]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:56 smithi196 ceph-mon[76349]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:56.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:56.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:56 smithi184 ceph-mon[100622]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:56.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:56.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:56 smithi184 ceph-mon[102844]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:58 smithi196 ceph-mon[76349]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:58.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:58 smithi184 ceph-mon[100622]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:58.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:58 smithi184 ceph-mon[102844]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:59.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:00 smithi196 ceph-mon[76349]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:00.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:00 smithi184 ceph-mon[100622]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:00.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:00 smithi184 ceph-mon[102844]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:01.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:01.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:01.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:02.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:02 smithi196 ceph-mon[76349]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:02.757 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:02 smithi184 ceph-mon[100622]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:02.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:02 smithi184 ceph-mon[102844]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:04 smithi196 ceph-mon[76349]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:04 smithi184 ceph-mon[100622]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:04 smithi184 ceph-mon[102844]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:05.526 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 6h - - 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 16s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 16s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 16s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:17:06.049 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 16s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 16s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 16s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 16s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 16s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 74.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:17:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:17:06.051 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:17:06.051 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:17:06.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:17:06.363 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:17:06.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:17:06.365 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:17:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:17:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:17:06.632 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[100622]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[100622]: from='client.70173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[100622]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[102844]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[102844]: from='client.70173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:06 smithi184 ceph-mon[102844]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:06 smithi196 ceph-mon[76349]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:06 smithi196 ceph-mon[76349]: from='client.70173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:06 smithi196 ceph-mon[76349]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:06.939 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:17:06.939 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:06.939 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:06.939 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:06.939 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:07.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:07 smithi196 ceph-mon[76349]: from='client.70185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/99740772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:07.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:07 smithi196 ceph-mon[76349]: from='client.80162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.605 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2366068843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:07.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[100622]: from='client.70185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/99740772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[100622]: from='client.80162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2366068843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[102844]: from='client.70185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/99740772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[102844]: from='client.80162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2366068843' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:08 smithi196 ceph-mon[76349]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:08.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:08 smithi184 ceph-mon[102844]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:08 smithi184 ceph-mon[100622]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:10 smithi196 ceph-mon[76349]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[100622]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[102844]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:17:12.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:11.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:11.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:11.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[102844]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[100622]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:12 smithi196 ceph-mon[76349]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:14.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:14 smithi184 ceph-mon[102844]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:14 smithi184 ceph-mon[100622]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:14 smithi196 ceph-mon[76349]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:16 smithi184 ceph-mon[102844]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:16 smithi184 ceph-mon[100622]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:16 smithi196 ceph-mon[76349]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:18 smithi184 ceph-mon[100622]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:18 smithi184 ceph-mon[102844]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:18 smithi196 ceph-mon[76349]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:20.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:20 smithi184 ceph-mon[102844]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:20 smithi184 ceph-mon[100622]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:20 smithi196 ceph-mon[76349]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:17:22.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:21.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:22.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:21.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:21.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:22 smithi184 ceph-mon[100622]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:22 smithi184 ceph-mon[102844]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:22 smithi196 ceph-mon[76349]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:24 smithi184 ceph-mon[100622]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:24 smithi184 ceph-mon[102844]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:24 smithi196 ceph-mon[76349]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:26 smithi184 ceph-mon[100622]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:26 smithi184 ceph-mon[102844]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:26 smithi196 ceph-mon[76349]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:28.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:28 smithi196 ceph-mon[76349]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:28.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:28 smithi184 ceph-mon[100622]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:28.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:28 smithi184 ceph-mon[102844]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[100622]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[102844]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:30 smithi196 ceph-mon[76349]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:17:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:31.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:31.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:31.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:32 smithi196 ceph-mon[76349]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[102844]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[100622]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:34 smithi184 ceph-mon[100622]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:34.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:34 smithi184 ceph-mon[102844]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:34 smithi196 ceph-mon[76349]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:36 smithi196 ceph-mon[76349]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:36 smithi184 ceph-mon[100622]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:37.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:36 smithi184 ceph-mon[102844]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:37.233 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 6h - - 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 6s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 6s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:17:37.773 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 6s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 6s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 6s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 6s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 6s ago 6h 74.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:17:37.774 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:17:37.775 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:17:37.775 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:17:37.775 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 6s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:17:38.086 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:17:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:17:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:17:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:17:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:17:38.088 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:17:38.368 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[100622]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[100622]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[100622]: from='client.70215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[100622]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/332487561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:38.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[102844]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[102844]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:38.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[102844]: from='client.70215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[102844]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/332487561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:38.715 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:17:38.715 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:38.716 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:38.716 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:38 smithi196 ceph-mon[76349]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:38 smithi196 ceph-mon[76349]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:38 smithi196 ceph-mon[76349]: from='client.70215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:38 smithi196 ceph-mon[76349]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/332487561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:39 smithi196 ceph-mon[76349]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:39.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/663436903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:39.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:39 smithi184 ceph-mon[100622]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:39.947 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/663436903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:39.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:39 smithi184 ceph-mon[102844]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:39.947 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/663436903' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:40 smithi196 ceph-mon[76349]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:40 smithi184 ceph-mon[100622]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:40 smithi184 ceph-mon[102844]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:41.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:41.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:41.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:42 smithi196 ceph-mon[76349]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:42 smithi184 ceph-mon[100622]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:43.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:42 smithi184 ceph-mon[102844]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:44.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:44 smithi184 ceph-mon[100622]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:44 smithi184 ceph-mon[102844]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:44 smithi196 ceph-mon[76349]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:46 smithi196 ceph-mon[76349]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:47.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:46 smithi184 ceph-mon[102844]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:46 smithi184 ceph-mon[100622]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:48 smithi196 ceph-mon[76349]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:48 smithi184 ceph-mon[100622]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:48 smithi184 ceph-mon[102844]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:50 smithi196 ceph-mon[76349]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[102844]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[100622]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:17:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:17:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:51.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:52.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:51.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:17:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:17:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:17:51.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:17:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[102844]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[100622]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:52 smithi196 ceph-mon[76349]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:17:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:54 smithi184 ceph-mon[100622]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:54 smithi184 ceph-mon[102844]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:54 smithi196 ceph-mon[76349]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:56 smithi184 ceph-mon[102844]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:56 smithi184 ceph-mon[100622]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:56 smithi196 ceph-mon[76349]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:58 smithi184 ceph-mon[100622]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:59.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:58 smithi184 ceph-mon[102844]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:58 smithi196 ceph-mon[76349]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:59.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:59.718 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:00 smithi184 ceph-mon[100622]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:00 smithi184 ceph-mon[102844]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:00 smithi196 ceph-mon[76349]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:18:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:01.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:01.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:01.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:02 smithi184 ceph-mon[100622]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:02 smithi184 ceph-mon[102844]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:02 smithi196 ceph-mon[76349]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:04 smithi184 ceph-mon[100622]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:04 smithi184 ceph-mon[102844]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:04 smithi196 ceph-mon[76349]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:06 smithi184 ceph-mon[100622]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:06 smithi184 ceph-mon[102844]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:06 smithi196 ceph-mon[76349]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:08 smithi196 ceph-mon[76349]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:09.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:08 smithi184 ceph-mon[100622]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:09.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:08 smithi184 ceph-mon[102844]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:09.014 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 6h - - 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 19s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 19s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 19s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:18:09.541 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 19s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 19s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 19s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 19s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 19s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 19s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:18:09.542 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:18:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:18:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:18:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:18:09.853 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:18:09.853 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:18:09.854 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:18:10.129 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:18:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:18:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:18:10.130 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:18:10.454 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:18:10.455 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:10.455 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:10.455 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:10.455 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='client.80210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/23641001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1733939489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='client.80210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/23641001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1733939489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='client.80210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/23641001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1733939489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:12.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:11 smithi184 ceph-mon[100622]: from='client.70269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:12.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:11 smithi184 ceph-mon[102844]: from='client.70269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:11.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:11.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:12.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:11.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:11 smithi196 ceph-mon[76349]: from='client.70269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:18:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:12 smithi196 ceph-mon[76349]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[100622]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[102844]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:18:13.811634409Z level=info msg="Completed cleanup jobs" duration=11.366513ms 2024-09-21T02:18:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:14 smithi196 ceph-mon[76349]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:14 smithi184 ceph-mon[100622]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:14 smithi184 ceph-mon[102844]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:16 smithi196 ceph-mon[76349]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:16 smithi184 ceph-mon[100622]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:17.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:16 smithi184 ceph-mon[102844]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:19.125 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:18 smithi184 ceph-mon[100622]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:19.125 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:18 smithi184 ceph-mon[102844]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:18 smithi196 ceph-mon[76349]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:20 smithi196 ceph-mon[76349]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:20 smithi184 ceph-mon[100622]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:21.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:20 smithi184 ceph-mon[102844]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:18:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:21.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:21.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:21.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:22 smithi196 ceph-mon[76349]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:22 smithi184 ceph-mon[100622]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:22 smithi184 ceph-mon[102844]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:24 smithi196 ceph-mon[76349]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:24 smithi184 ceph-mon[100622]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:25.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:24 smithi184 ceph-mon[102844]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:26 smithi196 ceph-mon[76349]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:26 smithi184 ceph-mon[100622]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:26 smithi184 ceph-mon[102844]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:28 smithi196 ceph-mon[76349]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:28 smithi184 ceph-mon[100622]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:28 smithi184 ceph-mon[102844]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:31.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:30 smithi196 ceph-mon[76349]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:31.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[100622]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[102844]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:31.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:31.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:31.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:32.200 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:18:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:32 smithi196 ceph-mon[76349]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[100622]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[102844]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:33.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:34 smithi196 ceph-mon[76349]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:34 smithi184 ceph-mon[100622]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:34 smithi184 ceph-mon[102844]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:36 smithi196 ceph-mon[76349]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:36 smithi184 ceph-mon[102844]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:36 smithi184 ceph-mon[100622]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:38 smithi196 ceph-mon[76349]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:38 smithi184 ceph-mon[100622]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:38 smithi184 ceph-mon[102844]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:40.746 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:18:41.031 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:40 smithi184 ceph-mon[100622]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:41.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:40 smithi184 ceph-mon[102844]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:40 smithi196 ceph-mon[76349]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 6h - - 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 9s ago 6h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:18:41.295 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 9s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 9s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 6h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 9s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 9s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 9s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 9s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 9s ago 6h 74.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:18:41.296 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:18:41.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:18:41.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:18:41.297 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:18:41.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:18:41.622 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:18:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:18:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:18:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:18:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:18:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:18:41.924 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:18:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:41 smithi196 ceph-mon[76349]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/921494293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:42.241 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:18:42.241 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:42.241 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:42.241 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:42.241 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:42.241 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-mon[100622]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:42.241 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/921494293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:42.242 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-mon[102844]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:42.242 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/921494293' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:42.242 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:18:42.242 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:41.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:42.242 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:41.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:42.242 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:41.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[100622]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[100622]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[100622]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3960648487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[102844]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[102844]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[102844]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3960648487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:42 smithi196 ceph-mon[76349]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:42 smithi196 ceph-mon[76349]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:42 smithi196 ceph-mon[76349]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3960648487' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:43 smithi196 ceph-mon[76349]: from='client.70305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:44.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:43 smithi184 ceph-mon[100622]: from='client.70305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:43 smithi184 ceph-mon[102844]: from='client.70305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:44 smithi196 ceph-mon[76349]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:44 smithi184 ceph-mon[100622]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:44 smithi184 ceph-mon[102844]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:46 smithi184 ceph-mon[100622]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:46 smithi184 ceph-mon[102844]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:46 smithi196 ceph-mon[76349]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:48 smithi184 ceph-mon[100622]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:48 smithi184 ceph-mon[102844]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:48 smithi196 ceph-mon[76349]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:51.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:50 smithi196 ceph-mon[76349]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:51.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[100622]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[102844]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:18:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:18:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:51.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:51.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:18:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:18:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:18:51.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:18:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[100622]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[102844]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:52 smithi196 ceph-mon[76349]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:18:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:54 smithi184 ceph-mon[100622]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:54 smithi184 ceph-mon[102844]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:54 smithi196 ceph-mon[76349]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:56 smithi184 ceph-mon[100622]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:56 smithi184 ceph-mon[102844]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:56 smithi196 ceph-mon[76349]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:58 smithi184 ceph-mon[100622]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:58 smithi184 ceph-mon[102844]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:58 smithi196 ceph-mon[76349]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:18:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:18:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:18:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:00 smithi196 ceph-mon[76349]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:00 smithi184 ceph-mon[100622]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:00 smithi184 ceph-mon[102844]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:02.002 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:01 smithi184 ceph-mon[100622]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:02.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:01 smithi184 ceph-mon[102844]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:01.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:01.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:02.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:01.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:02.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:19:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:01 smithi196 ceph-mon[76349]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:04 smithi184 ceph-mon[100622]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:04 smithi184 ceph-mon[102844]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:04 smithi196 ceph-mon[76349]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:06 smithi184 ceph-mon[100622]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:06 smithi184 ceph-mon[102844]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:06 smithi196 ceph-mon[76349]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:08 smithi184 ceph-mon[100622]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:08 smithi184 ceph-mon[102844]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:08 smithi196 ceph-mon[76349]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:10.733 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:10 smithi196 ceph-mon[76349]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:10.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:10 smithi184 ceph-mon[100622]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:10.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:10 smithi184 ceph-mon[102844]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:11.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:11.851 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:19:12.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:11.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:11.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:11.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:12.532 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:19:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[100622]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[102844]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:12 smithi196 ceph-mon[76349]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:12.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:12.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:13.080 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:19:13.080 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T02:19:13.080 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 6h - - 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 2s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 0s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 2s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 0s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 2s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 2s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 0s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 2s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 2s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 0s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 2s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 2s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 2s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:19:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 2s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 0s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 0s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 0s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 0s ago 6h 75.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:19:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:19:13.397 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:13.398 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:19:13.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:19:13.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:19:13.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:19:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:13 smithi196 ceph-mon[76349]: from='client.80282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:13 smithi196 ceph-mon[76349]: from='client.80288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/681442758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[100622]: from='client.80282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[100622]: from='client.80288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/681442758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[102844]: from='client.80282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[102844]: from='client.80288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:13.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/681442758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:13.669 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:19:13.670 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:19:13.977 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:19:13.977 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:13.977 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:14 smithi196 ceph-mon[76349]: from='client.70329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:14 smithi196 ceph-mon[76349]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:14 smithi196 ceph-mon[76349]: from='client.80306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/298940246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[100622]: from='client.70329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[100622]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[100622]: from='client.80306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/298940246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[102844]: from='client.70329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[102844]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[102844]: from='client.80306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/298940246' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:16 smithi196 ceph-mon[76349]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:16 smithi184 ceph-mon[100622]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:16 smithi184 ceph-mon[102844]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:18 smithi196 ceph-mon[76349]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:18 smithi184 ceph-mon[100622]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:19.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:18 smithi184 ceph-mon[102844]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:20 smithi196 ceph-mon[76349]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:20 smithi184 ceph-mon[100622]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:20 smithi184 ceph-mon[102844]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:22.149 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:19:22.149 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:21.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:22.149 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:21.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:22.149 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:21.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:22.696 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:22 smithi184 ceph-mon[100622]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:19:22.697 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:22 smithi184 ceph-mon[102844]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:22 smithi196 ceph-mon[76349]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:19:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:24 smithi196 ceph-mon[76349]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:24 smithi184 ceph-mon[100622]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:24 smithi184 ceph-mon[102844]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:26 smithi196 ceph-mon[76349]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:26 smithi184 ceph-mon[100622]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:26 smithi184 ceph-mon[102844]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:28 smithi196 ceph-mon[76349]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:28 smithi184 ceph-mon[100622]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:28 smithi184 ceph-mon[102844]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:30 smithi196 ceph-mon[76349]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:30 smithi184 ceph-mon[100622]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:30 smithi184 ceph-mon[102844]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:31.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:31.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:31.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:31.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:31.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:32.004 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:31.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:19:32.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:32 smithi196 ceph-mon[76349]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:32.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:32.825 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[100622]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[102844]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:34 smithi184 ceph-mon[100622]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:34 smithi184 ceph-mon[102844]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:34 smithi196 ceph-mon[76349]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:36 smithi184 ceph-mon[100622]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:36 smithi184 ceph-mon[102844]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:36 smithi196 ceph-mon[76349]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:38 smithi184 ceph-mon[100622]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:38 smithi184 ceph-mon[102844]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:38 smithi196 ceph-mon[76349]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:40 smithi184 ceph-mon[100622]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:40 smithi184 ceph-mon[102844]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:40 smithi196 ceph-mon[76349]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:19:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:41.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:41.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:41.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:43.008 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:42 smithi184 ceph-mon[100622]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:43.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:42 smithi184 ceph-mon[102844]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:42 smithi196 ceph-mon[76349]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:44.284 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:19:44.817 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 12s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 12s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 12s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 12s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 12s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 12s ago 6h 75.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:19:44.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:19:44.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:19:44.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:19:44.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 12s ago 6h 91.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:19:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:44 smithi184 ceph-mon[100622]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:45.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:44 smithi184 ceph-mon[102844]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:45.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:19:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:19:45.130 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:44 smithi196 ceph-mon[76349]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:45.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:19:45.402 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:19:45.402 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:19:45.402 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:19:45.402 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:19:45.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:19:45.711 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:19:45.712 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:45.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:45.712 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:45.712 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[100622]: from='client.80318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[100622]: from='client.80324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[100622]: from='client.80330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/601949327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2769325355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[102844]: from='client.80318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[102844]: from='client.80324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[102844]: from='client.80330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/601949327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2769325355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:45 smithi196 ceph-mon[76349]: from='client.80318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:45 smithi196 ceph-mon[76349]: from='client.80324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:45 smithi196 ceph-mon[76349]: from='client.80330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/601949327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2769325355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[100622]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[100622]: from='client.70377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[102844]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:46 smithi184 ceph-mon[102844]: from='client.70377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:46 smithi196 ceph-mon[76349]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:46 smithi196 ceph-mon[76349]: from='client.70377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:48 smithi184 ceph-mon[100622]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:49.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:48 smithi184 ceph-mon[102844]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:48 smithi196 ceph-mon[76349]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:50 smithi184 ceph-mon[100622]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:50 smithi184 ceph-mon[102844]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:50 smithi196 ceph-mon[76349]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.138 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:19:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:51.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:51.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:19:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:19:51.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:19:53.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:52 smithi196 ceph-mon[76349]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:53.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[100622]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[102844]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:19:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:54 smithi196 ceph-mon[76349]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:54 smithi184 ceph-mon[100622]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:55.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:54 smithi184 ceph-mon[102844]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:56 smithi196 ceph-mon[76349]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:56 smithi184 ceph-mon[100622]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:56 smithi184 ceph-mon[102844]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:58 smithi196 ceph-mon[76349]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:58 smithi184 ceph-mon[100622]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:58 smithi184 ceph-mon[102844]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:19:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:00 smithi196 ceph-mon[76349]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:00 smithi184 ceph-mon[100622]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:00 smithi184 ceph-mon[102844]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:20:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:01.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:01.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:01.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:02 smithi196 ceph-mon[76349]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:02 smithi184 ceph-mon[100622]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:02 smithi184 ceph-mon[102844]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:05.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:04 smithi184 ceph-mon[100622]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:05.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:04 smithi184 ceph-mon[102844]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:04 smithi196 ceph-mon[76349]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:06 smithi196 ceph-mon[76349]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:07.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:06 smithi184 ceph-mon[102844]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:06 smithi184 ceph-mon[100622]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:08 smithi184 ceph-mon[100622]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:08 smithi184 ceph-mon[102844]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:08 smithi196 ceph-mon[76349]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:10 smithi184 ceph-mon[102844]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:10 smithi184 ceph-mon[100622]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:10 smithi196 ceph-mon[76349]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[100622]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-mon[102844]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:11.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:11.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:11.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:11 smithi196 ceph-mon[76349]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:14.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:14 smithi196 ceph-mon[76349]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:14.729 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:14 smithi184 ceph-mon[100622]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:14 smithi184 ceph-mon[102844]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:16.010 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:20:16.542 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:20:16.542 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 6h - - 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 3s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 3s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:20:16.543 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 3s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 3s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 3s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 3s ago 6h 75.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:20:16.544 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 3s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:20:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:16 smithi184 ceph-mon[102844]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:16.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:16 smithi184 ceph-mon[100622]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:20:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:20:16.856 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:20:17.122 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:20:17.123 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:20:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:16 smithi196 ceph-mon[76349]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:17.429 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:17.429 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:17.429 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:17.429 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:17.430 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[100622]: from='client.70389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[100622]: from='client.80360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[100622]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/493650500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3120038772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[102844]: from='client.70389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[102844]: from='client.80360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[102844]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/493650500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3120038772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:17 smithi196 ceph-mon[76349]: from='client.70389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:17 smithi196 ceph-mon[76349]: from='client.80360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:17 smithi196 ceph-mon[76349]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/493650500' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3120038772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:18 smithi184 ceph-mon[100622]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:18 smithi184 ceph-mon[100622]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:18 smithi184 ceph-mon[102844]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:18 smithi184 ceph-mon[102844]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:18 smithi196 ceph-mon[76349]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:18 smithi196 ceph-mon[76349]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:20.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:20 smithi196 ceph-mon[76349]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:20 smithi184 ceph-mon[100622]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:21.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:20 smithi184 ceph-mon[102844]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:22.125 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:20:22.126 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:21.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:22.126 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:21.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:22.126 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:21.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:22 smithi184 ceph-mon[100622]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:22 smithi184 ceph-mon[102844]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:23.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:22 smithi196 ceph-mon[76349]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:24 smithi184 ceph-mon[100622]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:24 smithi184 ceph-mon[102844]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:24 smithi196 ceph-mon[76349]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:26 smithi184 ceph-mon[100622]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:26 smithi184 ceph-mon[102844]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:26 smithi196 ceph-mon[76349]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:28 smithi184 ceph-mon[100622]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:28 smithi184 ceph-mon[102844]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:28 smithi196 ceph-mon[76349]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:30.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:30 smithi184 ceph-mon[100622]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:30 smithi184 ceph-mon[102844]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:30 smithi196 ceph-mon[76349]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[102844]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:32 smithi184 ceph-mon[100622]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:31.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:31.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:31.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:32 smithi196 ceph-mon[76349]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:34.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:34.803 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:34 smithi196 ceph-mon[76349]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:34 smithi184 ceph-mon[100622]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:35.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:34 smithi184 ceph-mon[102844]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:37.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:36 smithi184 ceph-mon[100622]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:36 smithi184 ceph-mon[102844]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:36 smithi196 ceph-mon[76349]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:38 smithi184 ceph-mon[100622]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:39.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:38 smithi184 ceph-mon[102844]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:38 smithi196 ceph-mon[76349]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:40 smithi184 ceph-mon[100622]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:40 smithi184 ceph-mon[102844]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:40 smithi196 ceph-mon[76349]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:20:42.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:41.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:41.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:41.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:42 smithi184 ceph-mon[100622]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:42 smithi184 ceph-mon[102844]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:42 smithi196 ceph-mon[76349]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:44 smithi184 ceph-mon[100622]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:44 smithi184 ceph-mon[102844]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:44 smithi196 ceph-mon[76349]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:46 smithi196 ceph-mon[76349]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:47.254 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:46 smithi184 ceph-mon[100622]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:47.254 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:47.255 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:46 smithi184 ceph-mon[102844]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:47.255 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:47.721 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 6h - - 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 17s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 17s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 17s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 17s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 15s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 17s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:20:48.257 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 17s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 17s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 17s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 17s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 17s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 15s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 15s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 15s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 15s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 15s ago 6h 75.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:20:48.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 15s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:20:48.569 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:20:48.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:20:48.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:20:48.571 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:20:48.840 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:20:48.840 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:20:48.841 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[100622]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[100622]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4072837716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[102844]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[102844]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:48.842 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4072837716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:48 smithi196 ceph-mon[76349]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:48 smithi196 ceph-mon[76349]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4072837716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:49.150 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:49.150 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:49.151 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:49.151 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:49.151 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:49 smithi196 ceph-mon[76349]: from='client.70431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:49 smithi196 ceph-mon[76349]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:49 smithi196 ceph-mon[76349]: from='client.70449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2452095216' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[100622]: from='client.70431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[100622]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[100622]: from='client.70449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2452095216' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[102844]: from='client.70431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[102844]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[102844]: from='client.70449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2452095216' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:50 smithi196 ceph-mon[76349]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:51.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:50 smithi184 ceph-mon[100622]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:51.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:50 smithi184 ceph-mon[102844]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:52.232 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:20:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:20:52.232 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:51.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:52.233 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:51.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:20:52.233 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:20:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:20:51.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:20:52.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[100622]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:52 smithi184 ceph-mon[102844]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:52 smithi196 ceph-mon[76349]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:20:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:54 smithi196 ceph-mon[76349]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:54 smithi184 ceph-mon[100622]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:54 smithi184 ceph-mon[102844]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:56 smithi184 ceph-mon[100622]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:56 smithi184 ceph-mon[102844]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:56 smithi196 ceph-mon[76349]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:57.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:58 smithi184 ceph-mon[100622]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:58 smithi184 ceph-mon[102844]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:58 smithi196 ceph-mon[76349]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:00 smithi184 ceph-mon[100622]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:00 smithi184 ceph-mon[102844]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:00 smithi196 ceph-mon[76349]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:21:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:01.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:01.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:01.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:02 smithi184 ceph-mon[100622]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:03.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:02 smithi184 ceph-mon[102844]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:02 smithi196 ceph-mon[76349]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:04 smithi184 ceph-mon[100622]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:04 smithi184 ceph-mon[102844]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:04 smithi196 ceph-mon[76349]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:07.042 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:06 smithi184 ceph-mon[100622]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:07.042 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:07.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:06 smithi184 ceph-mon[102844]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:07.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:06 smithi196 ceph-mon[76349]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:08 smithi184 ceph-mon[100622]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:09.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:08 smithi184 ceph-mon[102844]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:08 smithi196 ceph-mon[76349]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:10 smithi184 ceph-mon[100622]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:10 smithi184 ceph-mon[102844]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:10 smithi196 ceph-mon[76349]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:11.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:11.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:11.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:12 smithi196 ceph-mon[76349]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:12.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[100622]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:12 smithi184 ceph-mon[102844]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:14.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[100622]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:14 smithi184 ceph-mon[102844]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:14 smithi196 ceph-mon[76349]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:16.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:16 smithi184 ceph-mon[100622]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:16 smithi184 ceph-mon[102844]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:16 smithi196 ceph-mon[76349]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:18 smithi184 ceph-mon[100622]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:18 smithi184 ceph-mon[102844]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:18 smithi196 ceph-mon[76349]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:19.450 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:21:19.980 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:21:19.980 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 6h - - 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 6s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 6s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 6s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 6s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:21:19.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 6s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 6s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 6s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:21:19.982 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 6s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:21:20.293 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:21:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[100622]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[100622]: from='client.70461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[100622]: from='client.70467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2040062367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[102844]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[102844]: from='client.70461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[102844]: from='client.70467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2040062367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:21:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:21:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:21:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:21:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:21:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:21:20.568 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:21:20.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:20 smithi196 ceph-mon[76349]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:20.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:20 smithi196 ceph-mon[76349]: from='client.70461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:20 smithi196 ceph-mon[76349]: from='client.70467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:20.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2040062367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:20.880 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:21:20.880 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:20.880 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:20.880 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:20.880 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[102844]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[102844]: from='client.70485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/532934909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[100622]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[100622]: from='client.70485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/532934909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:21 smithi196 ceph-mon[76349]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:21 smithi196 ceph-mon[76349]: from='client.70485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/532934909' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:21.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:21.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:21.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:22 smithi184 ceph-mon[100622]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:22 smithi184 ceph-mon[102844]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:22 smithi196 ceph-mon[76349]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:24 smithi184 ceph-mon[100622]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:24 smithi184 ceph-mon[102844]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:24 smithi196 ceph-mon[76349]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:26.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:26 smithi184 ceph-mon[100622]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:26.500 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:26.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:26 smithi184 ceph-mon[102844]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:26.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:26 smithi196 ceph-mon[76349]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:28 smithi184 ceph-mon[100622]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:28 smithi184 ceph-mon[102844]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:28 smithi196 ceph-mon[76349]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:29.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:30.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:30 smithi196 ceph-mon[76349]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:30.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:30 smithi184 ceph-mon[100622]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:30 smithi184 ceph-mon[102844]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:21:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:31.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:31.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:31.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:32.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[102844]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[100622]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:32 smithi196 ceph-mon[76349]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[100622]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:34 smithi184 ceph-mon[102844]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:34 smithi196 ceph-mon[76349]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:36 smithi196 ceph-mon[76349]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:36 smithi184 ceph-mon[100622]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:36 smithi184 ceph-mon[102844]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:38.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:38 smithi184 ceph-mon[100622]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:38 smithi184 ceph-mon[102844]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:38 smithi196 ceph-mon[76349]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:40 smithi184 ceph-mon[100622]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:40 smithi184 ceph-mon[102844]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:40 smithi196 ceph-mon[76349]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:21:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:41.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:41.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:41.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:42 smithi184 ceph-mon[100622]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:42 smithi184 ceph-mon[102844]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:42 smithi196 ceph-mon[76349]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:44 smithi184 ceph-mon[102844]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:44 smithi184 ceph-mon[100622]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:44 smithi196 ceph-mon[76349]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:46 smithi184 ceph-mon[100622]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:46 smithi184 ceph-mon[102844]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:46 smithi196 ceph-mon[76349]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:48 smithi184 ceph-mon[100622]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:48 smithi184 ceph-mon[102844]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:48 smithi196 ceph-mon[76349]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:50.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:50 smithi184 ceph-mon[102844]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:50 smithi184 ceph-mon[100622]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:50 smithi196 ceph-mon[76349]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:51.175 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 6h - - 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 0s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:21:51.714 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 0s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 0s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 0s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 0s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 0s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:21:51.715 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:21:51.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:21:51.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:21:51.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:51.864Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:51.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:21:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:21:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:21:51.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:21:52.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:21:52.029 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:21:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:21:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:21:52.297 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:21:52.298 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:21:52.610 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:21:52.610 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:52.611 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='client.80462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='client.80468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1177931268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1516923778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='client.80462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='client.80468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1177931268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1516923778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='client.80462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='client.80468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1177931268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:53.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1516923778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:53 smithi196 ceph-mon[76349]: from='client.80486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:53.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[100622]: from='client.80486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[102844]: from='client.80486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:21:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:54 smithi196 ceph-mon[76349]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:54 smithi184 ceph-mon[102844]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:54 smithi184 ceph-mon[100622]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:56 smithi196 ceph-mon[76349]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:56 smithi184 ceph-mon[100622]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:56 smithi184 ceph-mon[102844]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:58 smithi196 ceph-mon[76349]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:59.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:58 smithi184 ceph-mon[100622]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:58 smithi184 ceph-mon[102844]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:00 smithi184 ceph-mon[100622]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:00 smithi184 ceph-mon[102844]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:00 smithi196 ceph-mon[76349]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:22:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:01.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:01.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:01.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:02 smithi184 ceph-mon[100622]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:02 smithi184 ceph-mon[102844]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:02 smithi196 ceph-mon[76349]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:04 smithi184 ceph-mon[100622]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:04 smithi184 ceph-mon[102844]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:04 smithi196 ceph-mon[76349]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:06 smithi184 ceph-mon[100622]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:06 smithi184 ceph-mon[102844]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:06 smithi196 ceph-mon[76349]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:08 smithi184 ceph-mon[100622]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:08 smithi184 ceph-mon[102844]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:08.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:08 smithi196 ceph-mon[76349]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:10 smithi184 ceph-mon[100622]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:10 smithi184 ceph-mon[102844]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:10 smithi196 ceph-mon[76349]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:22:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:11.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:11.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:11.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[100622]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[102844]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:12 smithi196 ceph-mon[76349]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[100622]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[102844]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:14 smithi196 ceph-mon[76349]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:16.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:16 smithi196 ceph-mon[76349]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:16.763 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:16 smithi184 ceph-mon[100622]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:16.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:16 smithi184 ceph-mon[102844]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:18 smithi184 ceph-mon[100622]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:18 smithi184 ceph-mon[102844]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:18.775 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:18 smithi196 ceph-mon[76349]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:20.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:20 smithi196 ceph-mon[76349]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:20.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:20 smithi184 ceph-mon[100622]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:20 smithi184 ceph-mon[102844]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:22:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:21.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:21.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:21.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:22.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:22 smithi184 ceph-mon[100622]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:22 smithi184 ceph-mon[102844]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:22 smithi196 ceph-mon[76349]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:22.906 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:22:23.432 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 6h - - 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 6h - - 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 11s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 10s ago 6h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 10s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 10s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 11s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 10s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:22:23.433 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:22:23.434 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:22:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:22:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:22:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:22:23.765 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:22:23.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:23 smithi184 ceph-mon[100622]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:23.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:23 smithi184 ceph-mon[102844]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:23 smithi196 ceph-mon[76349]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:22:24.039 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:22:24.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:22:24.348 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:22:24.348 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:24.348 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:24.348 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:24.348 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[100622]: from='client.80504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[100622]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[100622]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1104710341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1225785158' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[102844]: from='client.80504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[102844]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[102844]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1104710341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1225785158' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:24 smithi196 ceph-mon[76349]: from='client.80504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:24 smithi196 ceph-mon[76349]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:24 smithi196 ceph-mon[76349]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1104710341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1225785158' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:25.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:25 smithi184 ceph-mon[100622]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:25.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:25 smithi184 ceph-mon[102844]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:25 smithi196 ceph-mon[76349]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:26 smithi184 ceph-mon[100622]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:26.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:26.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:26 smithi184 ceph-mon[102844]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:26 smithi196 ceph-mon[76349]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:28 smithi184 ceph-mon[100622]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:28 smithi184 ceph-mon[102844]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:28 smithi196 ceph-mon[76349]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:29.623 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:29.623 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:30 smithi184 ceph-mon[100622]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:30 smithi184 ceph-mon[102844]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:30 smithi196 ceph-mon[76349]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:31.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:31.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:31.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[102844]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[100622]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:32 smithi196 ceph-mon[76349]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[100622]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[102844]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:34 smithi196 ceph-mon[76349]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:36 smithi184 ceph-mon[100622]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:36.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:36 smithi184 ceph-mon[102844]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:36 smithi196 ceph-mon[76349]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:38 smithi184 ceph-mon[100622]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:38.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:38 smithi184 ceph-mon[102844]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:38.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:38 smithi196 ceph-mon[76349]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:40 smithi184 ceph-mon[100622]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:40.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:40 smithi184 ceph-mon[102844]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:40 smithi196 ceph-mon[76349]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:22:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:41.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:41.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:41.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:42 smithi184 ceph-mon[100622]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:42 smithi184 ceph-mon[102844]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:42 smithi196 ceph-mon[76349]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:44 smithi184 ceph-mon[102844]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:44.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:44 smithi184 ceph-mon[100622]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:44 smithi196 ceph-mon[76349]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:46 smithi196 ceph-mon[76349]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:46 smithi184 ceph-mon[100622]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:46 smithi184 ceph-mon[102844]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:48.844 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:48 smithi184 ceph-mon[100622]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:48.844 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:48 smithi184 ceph-mon[102844]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:48 smithi196 ceph-mon[76349]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:22:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:50 smithi196 ceph-mon[76349]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:50.955 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:50 smithi184 ceph-mon[100622]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:50.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:50 smithi184 ceph-mon[102844]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:22:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:51.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:51.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:22:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:22:51.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:52 smithi196 ceph-mon[76349]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[100622]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[102844]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:54.644 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:22:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[100622]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[102844]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:54 smithi196 ceph-mon[76349]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 6h - - 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 3s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 3s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:22:55.173 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 3s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 3s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 3s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 3s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 3s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 3s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 76.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:22:55.174 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:22:55.175 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:22:55.487 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:22:55.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:22:55.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:22:55.489 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:22:55.760 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:22:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:22:55.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[102844]: from='client.80534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[102844]: from='client.80540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/734300780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[100622]: from='client.80534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[100622]: from='client.80540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:55.762 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/734300780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:56.076 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:22:56.076 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:56.076 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:55 smithi196 ceph-mon[76349]: from='client.80534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:55 smithi196 ceph-mon[76349]: from='client.80540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/734300780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:57.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[102844]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[102844]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[102844]: from='client.80558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/800534029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[100622]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[100622]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[100622]: from='client.80558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/800534029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:56 smithi196 ceph-mon[76349]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:56 smithi196 ceph-mon[76349]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:56 smithi196 ceph-mon[76349]: from='client.80558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/800534029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:58.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:58 smithi196 ceph-mon[76349]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:58 smithi184 ceph-mon[100622]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:58 smithi184 ceph-mon[102844]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:00.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:00.067 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:00.070 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:00 smithi184 ceph-mon[100622]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:00 smithi184 ceph-mon[102844]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:00 smithi196 ceph-mon[76349]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:23:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:01.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:01.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:01.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:02 smithi184 ceph-mon[100622]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:02 smithi184 ceph-mon[102844]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:02 smithi196 ceph-mon[76349]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:04 smithi184 ceph-mon[100622]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:05.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:04 smithi184 ceph-mon[102844]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:04 smithi196 ceph-mon[76349]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:06 smithi184 ceph-mon[100622]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:06 smithi184 ceph-mon[102844]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:06 smithi196 ceph-mon[76349]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:09.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:08 smithi184 ceph-mon[100622]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:09.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:08 smithi184 ceph-mon[102844]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:08 smithi196 ceph-mon[76349]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:11.010 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:10 smithi184 ceph-mon[100622]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:11.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:10 smithi184 ceph-mon[102844]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:10 smithi196 ceph-mon[76349]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:12.155 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:23:12.155 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:11.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:12.155 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:11.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:12.155 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:11.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[100622]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[102844]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:12.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:12 smithi196 ceph-mon[76349]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[100622]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[102844]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:14 smithi196 ceph-mon[76349]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:16 smithi184 ceph-mon[100622]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:16 smithi184 ceph-mon[102844]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:16 smithi196 ceph-mon[76349]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:18 smithi184 ceph-mon[100622]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:18 smithi184 ceph-mon[102844]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:18 smithi196 ceph-mon[76349]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:20 smithi184 ceph-mon[100622]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:20 smithi184 ceph-mon[102844]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:20 smithi196 ceph-mon[76349]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:23:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:21.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:21.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:21.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:22 smithi184 ceph-mon[100622]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:22 smithi184 ceph-mon[102844]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:22 smithi196 ceph-mon[76349]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:24 smithi196 ceph-mon[76349]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:24 smithi184 ceph-mon[100622]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:24 smithi184 ceph-mon[102844]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:26.374 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 6h - - 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 13s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:23:26.902 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 13s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 13s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 13s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 13s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 13s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:23:26.903 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 13s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:23:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 6h 76.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:23:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:23:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:23:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:23:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 13s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:23:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:26 smithi196 ceph-mon[76349]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:27.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:27.213 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:23:27.213 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:23:27.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:23:27.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:23:27.214 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:23:27.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:26 smithi184 ceph-mon[102844]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:27.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:27.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:26 smithi184 ceph-mon[100622]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:27.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:23:27.480 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:23:27.481 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:23:27.790 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:23:27.790 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:27.790 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[100622]: from='client.70605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[100622]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[100622]: from='client.70617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/982775261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1392111932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[102844]: from='client.70605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[102844]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[102844]: from='client.70617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/982775261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1392111932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:27 smithi196 ceph-mon[76349]: from='client.70605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:27 smithi196 ceph-mon[76349]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:27 smithi196 ceph-mon[76349]: from='client.70617 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/982775261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1392111932' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:28 smithi196 ceph-mon[76349]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:28 smithi196 ceph-mon[76349]: from='client.70629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:29.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:28 smithi184 ceph-mon[100622]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:29.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:28 smithi184 ceph-mon[100622]: from='client.70629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:29.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:28 smithi184 ceph-mon[102844]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:29.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:28 smithi184 ceph-mon[102844]: from='client.70629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:30.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:30.283 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:30 smithi184 ceph-mon[100622]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:30 smithi184 ceph-mon[102844]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:30 smithi196 ceph-mon[76349]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:32.006 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:31.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:32.007 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:31.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:32.007 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:31.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:32.260 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:23:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:32 smithi196 ceph-mon[76349]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[100622]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[102844]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[100622]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[102844]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:34 smithi196 ceph-mon[76349]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:36 smithi184 ceph-mon[102844]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:36 smithi184 ceph-mon[100622]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:36 smithi196 ceph-mon[76349]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:39.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:38 smithi196 ceph-mon[76349]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:38 smithi184 ceph-mon[100622]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:38 smithi184 ceph-mon[102844]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:40 smithi184 ceph-mon[100622]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:40 smithi184 ceph-mon[102844]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:40 smithi196 ceph-mon[76349]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:23:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:41.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:41.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:41.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:42 smithi184 ceph-mon[100622]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:42 smithi184 ceph-mon[102844]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:42 smithi196 ceph-mon[76349]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:44 smithi184 ceph-mon[100622]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:44 smithi184 ceph-mon[102844]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:44 smithi196 ceph-mon[76349]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:46 smithi184 ceph-mon[100622]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:46 smithi184 ceph-mon[102844]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:46 smithi196 ceph-mon[76349]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:48 smithi184 ceph-mon[100622]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:48 smithi184 ceph-mon[102844]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:48 smithi196 ceph-mon[76349]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:51.164 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:50 smithi184 ceph-mon[100622]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:51.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:50 smithi184 ceph-mon[102844]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:51.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:50 smithi196 ceph-mon[76349]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:51.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:51.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:23:52.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:23:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:23:51.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:23:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:23:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:23:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[100622]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[102844]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:52 smithi196 ceph-mon[76349]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:53.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:53.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[100622]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[102844]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:54 smithi196 ceph-mon[76349]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:56 smithi184 ceph-mon[100622]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:56 smithi184 ceph-mon[102844]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:56 smithi196 ceph-mon[76349]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:58.085 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 6h - - 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 6s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 6s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 6s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:23:58.612 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 6s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 6s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 6s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 6s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 6s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 6s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 6s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 6h 76.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:23:58.613 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:23:58.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:23:58.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:58.923 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:23:58.924 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:23:59.194 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:23:59.194 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:23:59.194 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:23:59.195 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:23:59.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:58 smithi184 ceph-mon[100622]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:59.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1965468633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:59.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:58 smithi184 ceph-mon[102844]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:59.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1965468633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:58 smithi196 ceph-mon[76349]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1965468633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:59.505 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:23:59.505 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:59.505 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:59.505 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:59.505 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:59 smithi196 ceph-mon[76349]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:59 smithi196 ceph-mon[76349]: from='client.70647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:59 smithi196 ceph-mon[76349]: from='client.70653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:23:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3301525590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[100622]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[100622]: from='client.70647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[100622]: from='client.70653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3301525590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[102844]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[102844]: from='client.70647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[102844]: from='client.70653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:23:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3301525590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:01.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:00 smithi196 ceph-mon[76349]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:01.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:00 smithi196 ceph-mon[76349]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:00 smithi184 ceph-mon[100622]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:00 smithi184 ceph-mon[100622]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:00 smithi184 ceph-mon[102844]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:00 smithi184 ceph-mon[102844]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:24:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:01.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:01.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:01.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:02 smithi196 ceph-mon[76349]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:02 smithi184 ceph-mon[100622]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:03.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:02 smithi184 ceph-mon[102844]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:04 smithi184 ceph-mon[100622]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:04 smithi184 ceph-mon[102844]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:04 smithi196 ceph-mon[76349]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:06 smithi184 ceph-mon[100622]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:06 smithi184 ceph-mon[102844]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:06 smithi196 ceph-mon[76349]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:09.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:08 smithi184 ceph-mon[100622]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:24:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:08 smithi184 ceph-mon[102844]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:24:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:08 smithi196 ceph-mon[76349]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:24:10.848 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:10 smithi184 ceph-mon[100622]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:24:10.849 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:10 smithi184 ceph-mon[102844]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:24:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:10 smithi196 ceph-mon[76349]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:24:12.123 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:24:12.124 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:11.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:12.124 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:11.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:12.124 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:11.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:12 smithi196 ceph-mon[76349]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[100622]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:13.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[102844]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:13.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:13.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:14 smithi196 ceph-mon[76349]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[100622]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[102844]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:16 smithi196 ceph-mon[76349]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:24:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:16 smithi184 ceph-mon[100622]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:24:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:16 smithi184 ceph-mon[102844]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:24:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:18 smithi196 ceph-mon[76349]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:24:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:18 smithi184 ceph-mon[100622]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:24:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:18 smithi184 ceph-mon[102844]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 384 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:24:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:20 smithi196 ceph-mon[76349]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T02:24:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:20 smithi184 ceph-mon[100622]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T02:24:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:20 smithi184 ceph-mon[102844]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 388 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T02:24:22.176 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:24:22.176 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:21.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:22.176 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:21.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:22.176 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:21.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:23.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:22 smithi196 ceph-mon[76349]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:24:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:22 smithi184 ceph-mon[100622]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:24:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:22 smithi184 ceph-mon[102844]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:24:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:24 smithi196 ceph-mon[76349]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:24:25.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:24 smithi184 ceph-mon[100622]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:24:25.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:24 smithi184 ceph-mon[102844]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:24:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:26 smithi196 ceph-mon[76349]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T02:24:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:26 smithi184 ceph-mon[100622]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T02:24:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:26 smithi184 ceph-mon[102844]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T02:24:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:28 smithi196 ceph-mon[76349]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:28 smithi184 ceph-mon[100622]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:28 smithi184 ceph-mon[102844]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 371 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:29.838 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:24:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 6h - - 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 6h - - 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 17s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 16s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:24:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 17s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 16s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 17s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 17s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 16s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 17s ago 6h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 17s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 16s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 17s ago 6h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 17s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 17s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 17s ago 6h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 16s ago 6h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 16s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:24:30.382 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 16s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 16s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 16s ago 6h 76.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:24:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 16s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:24:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:24:30.692 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:24:30.961 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:24:30.962 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:24:30.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[100622]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:24:30.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[100622]: from='client.70677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:30.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2152279330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:30.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[102844]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:24:30.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[102844]: from='client.70677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:30.963 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2152279330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:30 smithi196 ceph-mon[76349]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:24:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:30 smithi196 ceph-mon[76349]: from='client.70677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2152279330' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:31.272 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:24:31.272 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:31.272 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:31 smithi196 ceph-mon[76349]: from='client.80648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:31 smithi196 ceph-mon[76349]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:31 smithi196 ceph-mon[76349]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/398760713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[100622]: from='client.80648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[100622]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[100622]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/398760713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[102844]: from='client.80648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[102844]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[102844]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/398760713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:31.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:31.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:31.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:32 smithi196 ceph-mon[76349]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:24:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[100622]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:24:33.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[102844]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:24:33.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:33.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:34 smithi196 ceph-mon[76349]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[100622]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[102844]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:36 smithi196 ceph-mon[76349]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T02:24:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:36 smithi184 ceph-mon[100622]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T02:24:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:36 smithi184 ceph-mon[102844]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T02:24:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:38 smithi196 ceph-mon[76349]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T02:24:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:38 smithi184 ceph-mon[100622]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T02:24:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:38 smithi184 ceph-mon[102844]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 359 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T02:24:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:40 smithi196 ceph-mon[76349]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:24:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:40 smithi184 ceph-mon[100622]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:24:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:40 smithi184 ceph-mon[102844]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:24:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:24:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:41.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:41.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:41.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:42 smithi196 ceph-mon[76349]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:24:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:42 smithi184 ceph-mon[100622]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:24:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:42 smithi184 ceph-mon[102844]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:24:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:44 smithi196 ceph-mon[76349]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T02:24:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:44 smithi184 ceph-mon[100622]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T02:24:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:44 smithi184 ceph-mon[102844]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T02:24:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:46 smithi196 ceph-mon[76349]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:24:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:46 smithi184 ceph-mon[100622]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:24:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:46 smithi184 ceph-mon[102844]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T02:24:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:48 smithi196 ceph-mon[76349]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:24:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:48 smithi184 ceph-mon[100622]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:24:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:48 smithi184 ceph-mon[102844]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:24:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:50 smithi196 ceph-mon[76349]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:24:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:50 smithi184 ceph-mon[100622]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:24:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:50 smithi184 ceph-mon[102844]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:24:52.222 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:24:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:24:52.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:51.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:52.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:51.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:24:52.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:24:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:24:51.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[100622]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[102844]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:52 smithi196 ceph-mon[76349]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:24:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:54 smithi196 ceph-mon[76349]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[100622]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[102844]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:55.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:24:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:56 smithi196 ceph-mon[76349]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:56 smithi184 ceph-mon[100622]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:56 smithi184 ceph-mon[102844]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:58 smithi196 ceph-mon[76349]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:58 smithi184 ceph-mon[100622]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:58 smithi184 ceph-mon[102844]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:00 smithi196 ceph-mon[76349]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:00 smithi184 ceph-mon[100622]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:00 smithi184 ceph-mon[102844]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:01.566 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 6h - - 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 9s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 7s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 9s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 7s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:25:02.091 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 9s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 9s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 9s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 7s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 9s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 9s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 9s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 9s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:25:02.092 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:25:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 7s ago 6h 76.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:25:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:25:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:25:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:25:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:25:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:25:02.297 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:01.880Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:01.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:01.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:25:02.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:25:02.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:02.403 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:25:02.403 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:25:02.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:25:02.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:25:02.672 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:25:02.673 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:25:02.673 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:25:02.673 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:25:02.673 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:25:02.984 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:25:02.984 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:02.984 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:02.985 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:02.985 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[100622]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[100622]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[100622]: from='client.80684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2153209828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[102844]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[102844]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:02.985 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[102844]: from='client.80684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:02.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2153209828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:03.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:02 smithi196 ceph-mon[76349]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:03.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:02 smithi196 ceph-mon[76349]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:03.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:02 smithi196 ceph-mon[76349]: from='client.80684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:03.030 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2153209828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:03 smithi196 ceph-mon[76349]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:03 smithi196 ceph-mon[76349]: from='client.70737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3919507402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[102844]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[102844]: from='client.70737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3919507402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:04.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[100622]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[100622]: from='client.70737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:04.301 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3919507402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:04 smithi184 ceph-mon[102844]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:04 smithi184 ceph-mon[100622]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:04 smithi196 ceph-mon[76349]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:06 smithi184 ceph-mon[100622]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:06 smithi184 ceph-mon[102844]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:06 smithi196 ceph-mon[76349]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:08 smithi184 ceph-mon[100622]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:08 smithi184 ceph-mon[102844]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:08 smithi196 ceph-mon[76349]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:10 smithi184 ceph-mon[100622]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:10 smithi184 ceph-mon[102844]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:10 smithi196 ceph-mon[76349]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:25:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:11.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:11.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:11.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:13.017 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:12 smithi184 ceph-mon[100622]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:13.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:12 smithi184 ceph-mon[102844]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:12 smithi196 ceph-mon[76349]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.141 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[100622]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:15.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:15.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.196 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[102844]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:15.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:15.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:14 smithi196 ceph-mon[76349]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:16 smithi184 ceph-mon[100622]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:16 smithi184 ceph-mon[102844]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:16 smithi196 ceph-mon[76349]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:18 smithi184 ceph-mon[100622]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:18 smithi184 ceph-mon[102844]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:18 smithi196 ceph-mon[76349]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:20 smithi184 ceph-mon[100622]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:20 smithi184 ceph-mon[102844]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:20 smithi196 ceph-mon[76349]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:25:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:21.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:21.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:21.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:22 smithi184 ceph-mon[100622]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:22 smithi184 ceph-mon[102844]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:23.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:22 smithi196 ceph-mon[76349]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:25.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:24 smithi196 ceph-mon[76349]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:24 smithi184 ceph-mon[100622]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:24 smithi184 ceph-mon[102844]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:26 smithi184 ceph-mon[100622]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:26 smithi184 ceph-mon[102844]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:26 smithi196 ceph-mon[76349]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:28 smithi184 ceph-mon[100622]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:25:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:28 smithi184 ceph-mon[102844]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:25:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:28 smithi196 ceph-mon[76349]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:25:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:30 smithi184 ceph-mon[100622]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:30 smithi184 ceph-mon[102844]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:30 smithi196 ceph-mon[76349]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:25:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:31.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:31.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:31.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:33.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:32 smithi184 ceph-mon[100622]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:33.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:32 smithi184 ceph-mon[102844]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:33.292 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:25:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:32 smithi196 ceph-mon[76349]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:33.861 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:25:33.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 6h - - 2024-09-21T02:25:33.861 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 6h - - 2024-09-21T02:25:33.861 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 0s ago 6h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 19s ago 6h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 0s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 19s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 0s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 0s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 19s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 0s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 0s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 19s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 19s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:25:33.862 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 19s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 19s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 19s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 19s ago 6h 76.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:25:33.863 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 19s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:25:34.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:25:34.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:25:34.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='client.80720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[102844]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='client.80720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:34 smithi184 ceph-mon[100622]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='client.80720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:34 smithi196 ceph-mon[76349]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:34.472 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:25:34.499 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:25:34.500 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:25:34.500 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:25:34.500 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:25:34.500 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:25:34.800 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:25:34.800 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:34.800 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:34.800 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1622090169' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='client.70773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2195539596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1622090169' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='client.70773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2195539596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:35.515 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1622090169' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:35.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='client.70773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:35.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:35.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2195539596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:36.088 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:36 smithi184 ceph-mon[100622]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:36.088 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:36.088 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:36 smithi184 ceph-mon[102844]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:36.089 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:36 smithi196 ceph-mon[76349]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:38.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:38 smithi184 ceph-mon[100622]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:38 smithi184 ceph-mon[102844]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:38 smithi196 ceph-mon[76349]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:40.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:40 smithi184 ceph-mon[100622]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:40 smithi184 ceph-mon[102844]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:40 smithi196 ceph-mon[76349]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:25:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:41.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:41.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:41.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:42.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:42 smithi184 ceph-mon[100622]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:42 smithi184 ceph-mon[102844]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:42 smithi196 ceph-mon[76349]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:44 smithi196 ceph-mon[76349]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:44 smithi184 ceph-mon[100622]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:44 smithi184 ceph-mon[102844]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:46.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:46 smithi184 ceph-mon[100622]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:46 smithi184 ceph-mon[102844]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:46 smithi196 ceph-mon[76349]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:48 smithi184 ceph-mon[100622]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:48 smithi184 ceph-mon[102844]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:48 smithi196 ceph-mon[76349]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:50.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:50 smithi184 ceph-mon[100622]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:50 smithi184 ceph-mon[102844]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:50 smithi196 ceph-mon[76349]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:25:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:25:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:51.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:51.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:25:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:25:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:25:51.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:25:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:52 smithi196 ceph-mon[76349]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:52 smithi184 ceph-mon[100622]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:52 smithi184 ceph-mon[102844]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[100622]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:54 smithi184 ceph-mon[102844]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:54 smithi196 ceph-mon[76349]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:25:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:56 smithi184 ceph-mon[100622]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:56 smithi184 ceph-mon[102844]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:56 smithi196 ceph-mon[76349]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:58 smithi196 ceph-mon[76349]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:58 smithi184 ceph-mon[100622]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:58 smithi184 ceph-mon[102844]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:25:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:25:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:25:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:00 smithi196 ceph-mon[76349]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:00 smithi184 ceph-mon[100622]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:00 smithi184 ceph-mon[102844]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:26:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:01.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:01.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:01.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:02 smithi196 ceph-mon[76349]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:26:03.210 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:02 smithi184 ceph-mon[100622]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:26:03.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:02 smithi184 ceph-mon[102844]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:26:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:04 smithi196 ceph-mon[76349]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:05.090 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:04 smithi184 ceph-mon[100622]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:05.090 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:04 smithi184 ceph-mon[102844]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:05.091 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 6h - - 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 10s ago 6h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 10s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 12s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 10s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 12s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:26:05.616 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 10s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 12s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 12s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 12s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 6h 77.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:26:05.617 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:26:05.928 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:26:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:26:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:26:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:26:05.929 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:26:06.197 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:26:06.198 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:26:06.519 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:26:06.519 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:06.519 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:06.519 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:06.519 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.80750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.70791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.80762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1849696687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:07.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1229214005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.80750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.70791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.80762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1849696687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1229214005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.80750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.70791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.80762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1849696687' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1229214005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:07 smithi196 ceph-mon[76349]: from='client.70809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:07 smithi184 ceph-mon[100622]: from='client.70809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:07 smithi184 ceph-mon[102844]: from='client.70809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:08 smithi196 ceph-mon[76349]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:08 smithi184 ceph-mon[100622]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:08 smithi184 ceph-mon[102844]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:10 smithi196 ceph-mon[76349]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:10 smithi184 ceph-mon[100622]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:10 smithi184 ceph-mon[102844]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:26:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:11.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:11.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:11.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:12 smithi196 ceph-mon[76349]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:12 smithi184 ceph-mon[100622]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:12 smithi184 ceph-mon[102844]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:14 smithi196 ceph-mon[76349]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[100622]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[102844]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:16.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:17.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:16 smithi184 ceph-mon[100622]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:17.198 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:17.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:16 smithi184 ceph-mon[102844]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:17.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:16 smithi196 ceph-mon[76349]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:19.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:18 smithi184 ceph-mon[102844]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:18 smithi184 ceph-mon[100622]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:18 smithi196 ceph-mon[76349]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:20 smithi184 ceph-mon[100622]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:20 smithi184 ceph-mon[102844]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:20 smithi196 ceph-mon[76349]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:26:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:21.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:21.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:21.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:22 smithi184 ceph-mon[100622]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:22 smithi184 ceph-mon[102844]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:22 smithi196 ceph-mon[76349]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:24 smithi196 ceph-mon[76349]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:24 smithi184 ceph-mon[100622]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:24 smithi184 ceph-mon[102844]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:26 smithi184 ceph-mon[100622]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:26 smithi184 ceph-mon[102844]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:26 smithi196 ceph-mon[76349]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:28 smithi184 ceph-mon[100622]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:28 smithi184 ceph-mon[102844]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:28 smithi196 ceph-mon[76349]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:30 smithi184 ceph-mon[100622]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:30 smithi184 ceph-mon[102844]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:30 smithi196 ceph-mon[76349]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:32] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:26:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:31.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:31.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:31.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:32 smithi184 ceph-mon[100622]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:32 smithi184 ceph-mon[102844]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:32 smithi196 ceph-mon[76349]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:34.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[100622]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:34 smithi184 ceph-mon[102844]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:34 smithi196 ceph-mon[76349]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[100622]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[102844]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:36 smithi196 ceph-mon[76349]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:36.823 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:26:37.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:37 smithi184 ceph-mon[100622]: from='client.70821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:37.166 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:37 smithi184 ceph-mon[102844]: from='client.70821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:37 smithi196 ceph-mon[76349]: from='client.70821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:37.393 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:26:37.393 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 6h - - 2024-09-21T02:26:37.393 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 6h - - 2024-09-21T02:26:37.393 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 3s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:26:37.393 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 2s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 3s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 2s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 3s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 3s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 2s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 3s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 3s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 3s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 3s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 2s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 2s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:26:37.394 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 2s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 2s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 2s ago 6h 77.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:26:37.395 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 2s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:26:37.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:26:37.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:26:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:26:38.003 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:26:38.228 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[100622]: from='client.70827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[100622]: from='client.70833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[100622]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1353855633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[102844]: from='client.70827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[102844]: from='client.70833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[102844]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:38.229 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1353855633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:38.316 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:26:38.317 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:38.317 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:38.317 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:38.317 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:38 smithi196 ceph-mon[76349]: from='client.70827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:38 smithi196 ceph-mon[76349]: from='client.70833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:38 smithi196 ceph-mon[76349]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1353855633' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:39 smithi196 ceph-mon[76349]: from='client.80810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/301944795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:39 smithi184 ceph-mon[102844]: from='client.80810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:39.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/301944795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:39 smithi184 ceph-mon[100622]: from='client.80810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/301944795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:40 smithi196 ceph-mon[76349]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:40.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:40 smithi184 ceph-mon[100622]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:40 smithi184 ceph-mon[102844]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:42] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:26:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:41.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:41.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:41.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:42 smithi196 ceph-mon[76349]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:42.897 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:42 smithi184 ceph-mon[100622]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:42.897 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:42 smithi184 ceph-mon[102844]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:44 smithi196 ceph-mon[76349]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:44 smithi184 ceph-mon[100622]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:44 smithi184 ceph-mon[102844]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:46 smithi196 ceph-mon[76349]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:46.997 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:46 smithi184 ceph-mon[100622]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:46.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:46.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:46 smithi184 ceph-mon[102844]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:46.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:48 smithi196 ceph-mon[76349]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:48 smithi184 ceph-mon[100622]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:48 smithi184 ceph-mon[102844]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:50 smithi196 ceph-mon[76349]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:50 smithi184 ceph-mon[100622]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:50 smithi184 ceph-mon[102844]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:26:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:26:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:51.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:51.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:26:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:26:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:26:51.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:26:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:52 smithi196 ceph-mon[76349]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:52 smithi184 ceph-mon[100622]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:52 smithi184 ceph-mon[102844]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:54 smithi196 ceph-mon[76349]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:54.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[100622]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[102844]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[100622]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[102844]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:56.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:26:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:56 smithi196 ceph-mon[76349]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:58.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:58 smithi184 ceph-mon[100622]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:58 smithi184 ceph-mon[102844]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:58 smithi196 ceph-mon[76349]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:59.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:00 smithi196 ceph-mon[76349]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:00 smithi184 ceph-mon[100622]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:00 smithi184 ceph-mon[102844]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:27:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:01.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:01.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:01.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:02 smithi196 ceph-mon[76349]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:02 smithi184 ceph-mon[100622]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:02 smithi184 ceph-mon[102844]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:04 smithi196 ceph-mon[76349]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:04 smithi184 ceph-mon[100622]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:04 smithi184 ceph-mon[102844]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:06 smithi196 ceph-mon[76349]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:07.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:06 smithi184 ceph-mon[100622]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:07.257 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:07.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:06 smithi184 ceph-mon[102844]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:07.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:08.612 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:27:08.948 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:08 smithi184 ceph-mon[100622]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:08.948 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:08 smithi184 ceph-mon[102844]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:08 smithi196 ceph-mon[76349]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 6h - - 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 13s ago 6h 92.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:27:09.143 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 13s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 13s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 13s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 13s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 13s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 13s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:27:09.144 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 6h 72.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:27:09.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:27:09.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:27:09.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:27:09.145 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 13s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:27:09.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:27:09.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:27:09.727 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:27:09.728 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:27:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[100622]: from='client.70857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[100622]: from='client.70863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3120852488' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[102844]: from='client.70857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[102844]: from='client.70863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3120852488' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:10.050 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:27:10.050 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:10.050 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:10.050 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:10.050 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:09 smithi196 ceph-mon[76349]: from='client.70857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:09 smithi196 ceph-mon[76349]: from='client.70863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3120852488' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:10 smithi196 ceph-mon[76349]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:10 smithi196 ceph-mon[76349]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:10 smithi196 ceph-mon[76349]: from='client.80846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1929248284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:11.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[102844]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[102844]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[102844]: from='client.80846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1929248284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[100622]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[100622]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[100622]: from='client.80846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1929248284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:27:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:11.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:11.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:11.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:12 smithi196 ceph-mon[76349]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:12 smithi184 ceph-mon[100622]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:12 smithi184 ceph-mon[102844]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[100622]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[102844]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:14 smithi196 ceph-mon[76349]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:16 smithi196 ceph-mon[76349]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[100622]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[102844]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:18.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:18 smithi184 ceph-mon[100622]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:18 smithi184 ceph-mon[102844]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:18 smithi196 ceph-mon[76349]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:20 smithi196 ceph-mon[76349]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:20 smithi184 ceph-mon[100622]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:20 smithi184 ceph-mon[102844]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:21.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:21.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:21.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:22 smithi196 ceph-mon[76349]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:22 smithi184 ceph-mon[100622]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:23.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:22 smithi184 ceph-mon[102844]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:24 smithi196 ceph-mon[76349]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:24 smithi184 ceph-mon[100622]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:24 smithi184 ceph-mon[102844]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:26 smithi196 ceph-mon[76349]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:26 smithi184 ceph-mon[100622]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:26 smithi184 ceph-mon[102844]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:28.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:28 smithi196 ceph-mon[76349]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:28 smithi184 ceph-mon[100622]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:28 smithi184 ceph-mon[102844]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:29.804 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:30 smithi196 ceph-mon[76349]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:30 smithi184 ceph-mon[100622]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:30 smithi184 ceph-mon[102844]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:27:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:31.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:31.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:31.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:32 smithi196 ceph-mon[76349]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:32 smithi184 ceph-mon[100622]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:32 smithi184 ceph-mon[102844]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:34 smithi196 ceph-mon[76349]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[100622]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[102844]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[102844]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[100622]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:36 smithi196 ceph-mon[76349]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:38 smithi196 ceph-mon[76349]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:38.909 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:38 smithi184 ceph-mon[100622]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:38.909 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:38 smithi184 ceph-mon[102844]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:40.341 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:27:40.870 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:27:40.870 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T02:27:40.870 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 5s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 6h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:27:40.871 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 5s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 5s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 5s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 5s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 73.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:27:40.872 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 5s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:27:40.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:40 smithi184 ceph-mon[100622]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:40.873 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:40 smithi184 ceph-mon[102844]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:40 smithi196 ceph-mon[76349]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:27:41.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:27:41.178 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:27:41.444 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:27:41.444 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:27:41.444 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:27:41.444 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:27:41.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:27:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:27:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:41.755 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:41.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[100622]: from='client.70893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.755 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[100622]: from='client.70899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[100622]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2780827684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[102844]: from='client.70893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[102844]: from='client.70899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[102844]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.756 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2780827684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:41 smithi196 ceph-mon[76349]: from='client.70893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:41 smithi196 ceph-mon[76349]: from='client.70899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:41 smithi196 ceph-mon[76349]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:41.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2780827684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:42.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:27:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:41.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:41.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:42.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:41.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:42 smithi196 ceph-mon[76349]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:42 smithi196 ceph-mon[76349]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/806434429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[100622]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[100622]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/806434429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[102844]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[102844]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/806434429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:44 smithi196 ceph-mon[76349]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:44 smithi184 ceph-mon[100622]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:44 smithi184 ceph-mon[102844]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:46 smithi196 ceph-mon[76349]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:46 smithi184 ceph-mon[100622]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:46 smithi184 ceph-mon[102844]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:48 smithi196 ceph-mon[76349]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:48 smithi184 ceph-mon[100622]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:48 smithi184 ceph-mon[102844]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:50 smithi196 ceph-mon[76349]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:50 smithi184 ceph-mon[100622]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:50 smithi184 ceph-mon[102844]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:27:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:27:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:51.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:51.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:27:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:27:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:27:51.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:27:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:52 smithi196 ceph-mon[76349]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:52 smithi184 ceph-mon[100622]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:52 smithi184 ceph-mon[102844]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[100622]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[102844]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:54 smithi196 ceph-mon[76349]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:56 smithi196 ceph-mon[76349]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[100622]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[102844]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:27:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:58.777 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:58 smithi184 ceph-mon[100622]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:58.777 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:58 smithi184 ceph-mon[102844]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:58 smithi196 ceph-mon[76349]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:59.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:59.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:00.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:00 smithi196 ceph-mon[76349]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:00 smithi184 ceph-mon[100622]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:00 smithi184 ceph-mon[102844]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:28:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:01.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:01.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:01.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:02 smithi196 ceph-mon[76349]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:03.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:02 smithi184 ceph-mon[102844]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:02 smithi184 ceph-mon[100622]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:04 smithi184 ceph-mon[100622]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:04 smithi184 ceph-mon[102844]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:04 smithi196 ceph-mon[76349]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:06 smithi196 ceph-mon[76349]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:06 smithi184 ceph-mon[100622]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:06 smithi184 ceph-mon[102844]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:08 smithi196 ceph-mon[76349]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:08 smithi184 ceph-mon[100622]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:08 smithi184 ceph-mon[102844]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:10.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:10 smithi196 ceph-mon[76349]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:10 smithi184 ceph-mon[100622]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:10 smithi184 ceph-mon[102844]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:12.051 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:28:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:28:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:11.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:11.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:11.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 389M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:28:12.579 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:28:12.580 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:28:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:12 smithi196 ceph-mon[76349]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:28:12.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:12.891 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:28:12.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:28:12.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:28:12.891 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:28:12.891 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:12 smithi184 ceph-mon[100622]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:12.891 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:12 smithi184 ceph-mon[102844]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:28:13.159 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:28:13.160 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:28:13.468 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:28:13.468 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:13.468 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[100622]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[100622]: from='client.80900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[100622]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1944333830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1092802267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[102844]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[102844]: from='client.80900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[102844]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1944333830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1092802267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:13.889 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:28:13.812208034Z level=info msg="Completed cleanup jobs" duration=12.745621ms 2024-09-21T02:28:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-mon[76349]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-mon[76349]: from='client.80900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-mon[76349]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1944333830' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1092802267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: from='client.70953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: from='client.70953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: from='client.70953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[100622]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[102844]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:16 smithi196 ceph-mon[76349]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:18 smithi184 ceph-mon[100622]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:18 smithi184 ceph-mon[102844]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:18 smithi196 ceph-mon[76349]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:28:20.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:20 smithi184 ceph-mon[100622]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:20.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:20 smithi184 ceph-mon[102844]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:21.080 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:20 smithi196 ceph-mon[76349]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:22.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:28:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:21.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:21.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:22.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:21.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:22 smithi184 ceph-mon[100622]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:22 smithi184 ceph-mon[102844]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:22 smithi196 ceph-mon[76349]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:24 smithi184 ceph-mon[100622]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:24 smithi184 ceph-mon[102844]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:24 smithi196 ceph-mon[76349]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:26 smithi184 ceph-mon[102844]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:26 smithi184 ceph-mon[100622]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:26 smithi196 ceph-mon[76349]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:28 smithi184 ceph-mon[100622]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:28 smithi184 ceph-mon[102844]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:28 smithi196 ceph-mon[76349]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:30.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:30 smithi196 ceph-mon[76349]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:30 smithi184 ceph-mon[100622]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:30 smithi184 ceph-mon[102844]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:28:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:31.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:31.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:31.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:32 smithi184 ceph-mon[100622]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:32 smithi184 ceph-mon[102844]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:32 smithi196 ceph-mon[76349]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[100622]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[102844]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:34 smithi196 ceph-mon[76349]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:36 smithi196 ceph-mon[76349]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[100622]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[102844]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:38 smithi196 ceph-mon[76349]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:38 smithi184 ceph-mon[100622]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:39.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:38 smithi184 ceph-mon[102844]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:41.114 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:40 smithi184 ceph-mon[100622]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:41.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:40 smithi184 ceph-mon[102844]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:40 smithi196 ceph-mon[76349]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:41.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:41.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:42.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:41.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:28:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:42 smithi196 ceph-mon[76349]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:42 smithi184 ceph-mon[100622]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:42 smithi184 ceph-mon[102844]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:43.768 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:28:44.298 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:28:44.299 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 8s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 8s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 8s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:28:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 8s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:44.610 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:28:44.611 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:28:44.879 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:28:44.879 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:28:44.880 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[100622]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[100622]: from='client.80930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3472549760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[102844]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[102844]: from='client.80930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:44.881 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3472549760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:44 smithi196 ceph-mon[76349]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:44 smithi196 ceph-mon[76349]: from='client.80930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3472549760' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:45.193 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:28:45.193 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:45.193 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:45.193 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:45.193 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:45 smithi196 ceph-mon[76349]: from='client.80936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:45 smithi196 ceph-mon[76349]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:45 smithi196 ceph-mon[76349]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/782461767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:46.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[102844]: from='client.80936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[102844]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[102844]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/782461767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[100622]: from='client.80936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[100622]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[100622]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/782461767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:46 smithi196 ceph-mon[76349]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:47.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:46 smithi184 ceph-mon[102844]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:46 smithi184 ceph-mon[100622]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:48 smithi196 ceph-mon[76349]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:48 smithi184 ceph-mon[100622]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:48 smithi184 ceph-mon[102844]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:50.995 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:50 smithi196 ceph-mon[76349]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:50 smithi184 ceph-mon[100622]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:50 smithi184 ceph-mon[102844]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:28:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:28:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:51.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:51.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:28:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:28:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:28:51.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:28:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:52 smithi196 ceph-mon[76349]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:52 smithi184 ceph-mon[100622]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:52 smithi184 ceph-mon[102844]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:54 smithi196 ceph-mon[76349]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[100622]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[102844]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[102844]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[100622]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:56 smithi196 ceph-mon[76349]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:28:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:58 smithi184 ceph-mon[100622]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:58 smithi184 ceph-mon[102844]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:58 smithi196 ceph-mon[76349]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:28:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:28:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:28:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:01.192 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:00 smithi184 ceph-mon[100622]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:01.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:00 smithi184 ceph-mon[102844]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:00 smithi196 ceph-mon[76349]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:02.163 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:29:02.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:01.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:02.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:01.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:02.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:01.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:02 smithi184 ceph-mon[100622]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:02 smithi184 ceph-mon[102844]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:02 smithi196 ceph-mon[76349]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:04 smithi184 ceph-mon[100622]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:04 smithi184 ceph-mon[102844]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:04 smithi196 ceph-mon[76349]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:06 smithi184 ceph-mon[100622]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:06 smithi184 ceph-mon[102844]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:06 smithi196 ceph-mon[76349]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:08 smithi184 ceph-mon[100622]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:08 smithi184 ceph-mon[102844]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:08 smithi196 ceph-mon[76349]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:10 smithi184 ceph-mon[100622]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:10 smithi184 ceph-mon[102844]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:11.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:10 smithi196 ceph-mon[76349]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:12 smithi184 ceph-mon[102844]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:12 smithi184 ceph-mon[100622]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:29:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:11.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:11.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:11.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:12.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:12 smithi196 ceph-mon[76349]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[100622]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[102844]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:14 smithi196 ceph-mon[76349]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:15.483 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 20s ago 6h - - 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 20s ago 6h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:29:16.013 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 20s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 20s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 20s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 20s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 20s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 20s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 20s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:29:16.014 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 20s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:29:16.015 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:29:16.015 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 20s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:29:16.015 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:29:16.015 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 20s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:29:16.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:29:16.328 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:29:16.329 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:29:16.599 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:29:16.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='client.80972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3008003600' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:29:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:16.907 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:16.907 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:16.907 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:16.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='client.80972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3008003600' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='client.80972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:16.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3008003600' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:17 smithi196 ceph-mon[76349]: from='client.80978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:17 smithi196 ceph-mon[76349]: from='client.80990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3596693648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[100622]: from='client.80978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[100622]: from='client.80990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3596693648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[102844]: from='client.80978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[102844]: from='client.80990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3596693648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:18 smithi196 ceph-mon[76349]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:18 smithi184 ceph-mon[100622]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:18 smithi184 ceph-mon[102844]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:20 smithi196 ceph-mon[76349]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:20 smithi184 ceph-mon[100622]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:20 smithi184 ceph-mon[102844]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:29:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:21.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:21.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:21.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:22.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:22 smithi184 ceph-mon[100622]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:22.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:22 smithi184 ceph-mon[102844]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:22 smithi196 ceph-mon[76349]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:24 smithi196 ceph-mon[76349]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:24 smithi184 ceph-mon[100622]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:24 smithi184 ceph-mon[102844]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:26 smithi196 ceph-mon[76349]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:26 smithi184 ceph-mon[100622]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:26 smithi184 ceph-mon[102844]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:28 smithi196 ceph-mon[76349]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:28 smithi184 ceph-mon[100622]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:28 smithi184 ceph-mon[102844]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:30 smithi196 ceph-mon[76349]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:30 smithi184 ceph-mon[102844]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:30 smithi184 ceph-mon[100622]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:29:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:31.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:31.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:31.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:32 smithi184 ceph-mon[100622]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:33.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:32 smithi184 ceph-mon[102844]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:33.124 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:32 smithi196 ceph-mon[76349]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:35.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:34 smithi196 ceph-mon[76349]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:35.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[102844]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[100622]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:36 smithi196 ceph-mon[76349]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[100622]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[102844]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:38 smithi196 ceph-mon[76349]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:38 smithi184 ceph-mon[100622]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:38 smithi184 ceph-mon[102844]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:40 smithi196 ceph-mon[76349]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:40 smithi184 ceph-mon[100622]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:40 smithi184 ceph-mon[102844]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:42.224 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:29:42.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:41.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:42.224 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:41.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:42.225 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:41.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:42 smithi196 ceph-mon[76349]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:42 smithi184 ceph-mon[100622]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:42 smithi184 ceph-mon[102844]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:44 smithi184 ceph-mon[100622]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:44 smithi184 ceph-mon[102844]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:44 smithi196 ceph-mon[76349]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:46 smithi196 ceph-mon[76349]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:47.205 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:46 smithi184 ceph-mon[100622]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:47.205 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:47.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:46 smithi184 ceph-mon[102844]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:47.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:47.206 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:29:47.731 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 6h - - 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 6h - - 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 13s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 13s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 13s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 13s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 13s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 13s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 13s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:29:47.732 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 13s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 13s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 74.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:29:47.733 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:29:48.046 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:29:48.046 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:29:48.046 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:29:48.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:48.046 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:29:48.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:29:48.048 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:29:48.318 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:29:48.318 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:29:48.319 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:29:48.626 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:29:48.626 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:48.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:48.626 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:48.626 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: from='client.81002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: from='client.71043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2625452426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1899733623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: from='client.81002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: from='client.71043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2625452426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1899733623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: from='client.81002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: from='client.71043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2625452426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1899733623' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:50.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:49 smithi184 ceph-mon[100622]: from='client.71061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:50.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:49 smithi184 ceph-mon[102844]: from='client.71061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:49 smithi196 ceph-mon[76349]: from='client.71061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:50 smithi184 ceph-mon[100622]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:50 smithi184 ceph-mon[102844]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:50 smithi196 ceph-mon[76349]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:29:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:29:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:51.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:51.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:29:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:29:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:29:51.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:29:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:52 smithi196 ceph-mon[76349]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:52 smithi184 ceph-mon[100622]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:52 smithi184 ceph-mon[102844]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[102844]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[100622]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:54 smithi196 ceph-mon[76349]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[100622]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[102844]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:56 smithi196 ceph-mon[76349]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:29:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:58 smithi184 ceph-mon[100622]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:58 smithi184 ceph-mon[102844]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:58 smithi196 ceph-mon[76349]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:00.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:29:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:00 smithi184 ceph-mon[100622]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:00 smithi184 ceph-mon[102844]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:00 smithi196 ceph-mon[76349]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:30:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:01.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:01.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:01.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:03.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:02 smithi184 ceph-mon[100622]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:03.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:02 smithi184 ceph-mon[102844]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:02 smithi196 ceph-mon[76349]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:04 smithi184 ceph-mon[100622]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:04 smithi184 ceph-mon[102844]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:04 smithi196 ceph-mon[76349]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:06 smithi184 ceph-mon[100622]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:06 smithi184 ceph-mon[102844]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:06 smithi196 ceph-mon[76349]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:08 smithi184 ceph-mon[100622]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:08 smithi184 ceph-mon[102844]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:08 smithi196 ceph-mon[76349]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:10 smithi184 ceph-mon[100622]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:10 smithi184 ceph-mon[102844]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:10 smithi196 ceph-mon[76349]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:30:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:11.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:11.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:11.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:13.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:12 smithi196 ceph-mon[76349]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:12 smithi184 ceph-mon[100622]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:12 smithi184 ceph-mon[102844]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:15.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:14 smithi196 ceph-mon[76349]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:15.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:15.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[100622]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[102844]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[100622]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[102844]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:16 smithi196 ceph-mon[76349]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:18.922 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:30:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:18 smithi184 ceph-mon[100622]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:18 smithi184 ceph-mon[102844]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:18 smithi196 ceph-mon[76349]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 6h - - 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 6h - - 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 2s ago 6h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 2s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 2s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:30:19.451 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 2s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 2s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 2s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 2s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 2s ago 6h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:30:19.452 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 2s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:30:19.759 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:30:19.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:30:19.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:30:20.031 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:30:20.032 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:30:20.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:19 smithi184 ceph-mon[100622]: from='client.81038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:20.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/343241586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:20.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:19 smithi184 ceph-mon[102844]: from='client.81038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:20.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/343241586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:20.343 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:20.344 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:20.344 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:20.344 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:19 smithi196 ceph-mon[76349]: from='client.81038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/343241586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[100622]: from='client.81044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[100622]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[100622]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3387939554' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[102844]: from='client.81044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[102844]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[102844]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3387939554' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:20 smithi196 ceph-mon[76349]: from='client.81044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:20 smithi196 ceph-mon[76349]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:20 smithi196 ceph-mon[76349]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3387939554' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:22.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:21 smithi184 ceph-mon[100622]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:22.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:21 smithi184 ceph-mon[102844]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:30:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:21.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:21.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:21.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:21 smithi196 ceph-mon[76349]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:22 smithi184 ceph-mon[100622]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:22 smithi184 ceph-mon[102844]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:22 smithi196 ceph-mon[76349]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:25.027 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:24 smithi184 ceph-mon[100622]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:25.027 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:24 smithi184 ceph-mon[102844]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:24 smithi196 ceph-mon[76349]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:27.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:26 smithi184 ceph-mon[102844]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:26 smithi184 ceph-mon[100622]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:26 smithi196 ceph-mon[76349]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:28 smithi184 ceph-mon[100622]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:28 smithi184 ceph-mon[102844]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:28 smithi196 ceph-mon[76349]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:30 smithi184 ceph-mon[100622]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:30 smithi184 ceph-mon[102844]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:30 smithi196 ceph-mon[76349]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:32.201 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:30:32.202 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:31.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:32.202 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:31.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:32.202 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:31.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:32 smithi184 ceph-mon[100622]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:32 smithi184 ceph-mon[102844]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:32 smithi196 ceph-mon[76349]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:35.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:34 smithi196 ceph-mon[76349]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:35.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.082 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[100622]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[102844]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[100622]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[102844]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:36 smithi196 ceph-mon[76349]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:38 smithi184 ceph-mon[100622]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:39 smithi184 ceph-mon[102844]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:38 smithi196 ceph-mon[76349]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:41 smithi184 ceph-mon[100622]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:41 smithi184 ceph-mon[102844]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:41 smithi196 ceph-mon[76349]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:30:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:41.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:41.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:41.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:43.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:43 smithi184 ceph-mon[100622]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:43.286 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:43 smithi184 ceph-mon[102844]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:43 smithi196 ceph-mon[76349]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:45 smithi184 ceph-mon[100622]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:45 smithi184 ceph-mon[102844]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:45 smithi196 ceph-mon[76349]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:47 smithi184 ceph-mon[100622]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:47 smithi184 ceph-mon[102844]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:47 smithi196 ceph-mon[76349]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:48 smithi184 ceph-mon[100622]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:48 smithi184 ceph-mon[102844]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:48 smithi196 ceph-mon[76349]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:50.628 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:30:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:50 smithi196 ceph-mon[76349]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:50 smithi184 ceph-mon[100622]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:51.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:50 smithi184 ceph-mon[102844]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:51.151 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:30:51.151 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 6h - - 2024-09-21T02:30:51.151 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 6h - - 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 93.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 16s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 16s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 16s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 16s ago 6h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 16s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 16s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 16s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:30:51.152 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 16s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 72.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:30:51.153 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:30:51.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:30:51.462 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:30:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:30:51.734 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:30:51.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[100622]: from='client.81074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[100622]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/10475021' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:51.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[102844]: from='client.81074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[102844]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/10475021' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:51 smithi196 ceph-mon[76349]: from='client.81074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:51 smithi196 ceph-mon[76349]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:51.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/10475021' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:51.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:51.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:30:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:30:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:30:51.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:30:52.043 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:52.043 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:52.043 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:52.043 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:52.043 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:30:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[100622]: from='client.71121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[100622]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[100622]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2897836990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[102844]: from='client.71121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[102844]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[102844]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2897836990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:52 smithi196 ceph-mon[76349]: from='client.71121 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:52 smithi196 ceph-mon[76349]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:52 smithi196 ceph-mon[76349]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2897836990' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:54 smithi184 ceph-mon[102844]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:54 smithi184 ceph-mon[100622]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:54 smithi196 ceph-mon[76349]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[100622]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[102844]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:56 smithi196 ceph-mon[76349]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:30:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:58 smithi184 ceph-mon[100622]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:58 smithi184 ceph-mon[102844]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:58 smithi196 ceph-mon[76349]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:00.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:01.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:00 smithi184 ceph-mon[100622]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:01.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:00 smithi184 ceph-mon[102844]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:00 smithi196 ceph-mon[76349]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:02.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:31:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:01.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:01.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:01.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:02 smithi184 ceph-mon[100622]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:02 smithi184 ceph-mon[102844]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:02 smithi196 ceph-mon[76349]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:04 smithi184 ceph-mon[100622]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:04 smithi184 ceph-mon[102844]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:04 smithi196 ceph-mon[76349]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:06.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:06 smithi184 ceph-mon[100622]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:06 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:06 smithi184 ceph-mon[102844]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:06 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:06 smithi196 ceph-mon[76349]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:06 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:08 smithi184 ceph-mon[100622]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:08 smithi184 ceph-mon[102844]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:08 smithi196 ceph-mon[76349]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:11.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:10 smithi184 ceph-mon[100622]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:10 smithi184 ceph-mon[102844]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:10 smithi196 ceph-mon[76349]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:31:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:11.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:11.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:11.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:12 smithi184 ceph-mon[100622]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:12 smithi184 ceph-mon[102844]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:12 smithi196 ceph-mon[76349]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:14 smithi184 ceph-mon[100622]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:14 smithi184 ceph-mon[102844]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:15.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:14 smithi196 ceph-mon[76349]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:15.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[100622]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[102844]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:16 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:16 smithi196 ceph-mon[76349]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:16 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:18 smithi196 ceph-mon[76349]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:18 smithi184 ceph-mon[100622]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:18 smithi184 ceph-mon[102844]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:20 smithi196 ceph-mon[76349]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:20 smithi184 ceph-mon[100622]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:20 smithi184 ceph-mon[102844]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:31:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:21.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:21.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:21.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:22.341 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:31:22.872 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:31:22.872 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 6h - - 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 6h - - 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 5s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:31:22.873 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 5s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 5s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 5s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 5s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:31:22.874 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 5s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:31:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:31:23.182 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:31:23.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:22 smithi184 ceph-mon[100622]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:23.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:22 smithi184 ceph-mon[102844]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:22 smithi196 ceph-mon[76349]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:23.452 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:31:23.452 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:31:23.452 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:31:23.452 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:31:23.452 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:31:23.453 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:31:23.776 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:31:23.776 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:23.776 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:23.776 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:23.777 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:24.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[100622]: from='client.71145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[100622]: from='client.81116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[100622]: from='client.81122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2954858749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3514332605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[102844]: from='client.71145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[102844]: from='client.81116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[102844]: from='client.81122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2954858749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3514332605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:23 smithi196 ceph-mon[76349]: from='client.71145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:23 smithi196 ceph-mon[76349]: from='client.81116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:23 smithi196 ceph-mon[76349]: from='client.81122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2954858749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3514332605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:24 smithi184 ceph-mon[100622]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:24 smithi184 ceph-mon[100622]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:24 smithi184 ceph-mon[102844]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:24 smithi184 ceph-mon[102844]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:24 smithi196 ceph-mon[76349]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:24 smithi196 ceph-mon[76349]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:26 smithi184 ceph-mon[100622]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:26 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:26 smithi184 ceph-mon[102844]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:26 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:27.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:26 smithi196 ceph-mon[76349]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:27.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:26 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:28 smithi184 ceph-mon[100622]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:28 smithi184 ceph-mon[102844]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:29.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:28 smithi196 ceph-mon[76349]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:30 smithi184 ceph-mon[100622]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:30 smithi184 ceph-mon[102844]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:30 smithi196 ceph-mon[76349]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:31:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:31.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:31.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:31.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:32 smithi184 ceph-mon[100622]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:32 smithi184 ceph-mon[102844]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:32 smithi196 ceph-mon[76349]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:34 smithi184 ceph-mon[100622]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:34 smithi184 ceph-mon[102844]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:34 smithi196 ceph-mon[76349]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:36 smithi196 ceph-mon[76349]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:36 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[100622]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[102844]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:36 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:39 smithi184 ceph-mon[100622]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:39 smithi184 ceph-mon[102844]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:39 smithi196 ceph-mon[76349]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:41 smithi184 ceph-mon[100622]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T02:31:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:41 smithi184 ceph-mon[102844]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T02:31:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:41 smithi196 ceph-mon[76349]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T02:31:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:31:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:41.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:41.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:41.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:43 smithi184 ceph-mon[102844]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:43 smithi184 ceph-mon[100622]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:43 smithi196 ceph-mon[76349]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:31:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:44 smithi196 ceph-mon[76349]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:44.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:44 smithi184 ceph-mon[100622]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:44 smithi184 ceph-mon[102844]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:31:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:45.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:46 smithi184 ceph-mon[100622]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:46 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:46 smithi184 ceph-mon[102844]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:46 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:46 smithi196 ceph-mon[76349]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:46 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:48.962 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:48 smithi184 ceph-mon[100622]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:48.962 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:48 smithi184 ceph-mon[102844]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:48 smithi196 ceph-mon[76349]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:31:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:50 smithi184 ceph-mon[100622]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:50 smithi184 ceph-mon[102844]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:50 smithi196 ceph-mon[76349]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:31:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:31:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:31:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:51.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:51.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:31:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:31:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:31:51.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:31:53.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:52 smithi184 ceph-mon[100622]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:52 smithi184 ceph-mon[102844]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:52 smithi196 ceph-mon[76349]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:54.071 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:31:54.601 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:31:54.601 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 6h - - 2024-09-21T02:31:54.601 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 6h - - 2024-09-21T02:31:54.601 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 19s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 94.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 19s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 19s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 19s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 19s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 19s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:31:54.602 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 19s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 19s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 19s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 73.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:31:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:31:54.909 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:31:54.909 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:31:54.909 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:31:54.910 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:31:54.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:54 smithi184 ceph-mon[100622]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:54.911 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:54 smithi184 ceph-mon[102844]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:54 smithi196 ceph-mon[76349]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:31:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:31:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:31:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:31:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:31:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:31:55.182 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:31:55.503 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:31:55.504 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:55.504 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:55.504 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:55.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='client.71181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='client.71187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1218389996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/554923052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='client.71181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='client.71187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1218389996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/554923052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='client.71181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='client.71187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1218389996' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/554923052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:56 smithi196 ceph-mon[76349]: from='client.71205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:56 smithi196 ceph-mon[76349]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:56 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[100622]: from='client.71205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[100622]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[102844]: from='client.71205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[102844]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:56 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:58 smithi196 ceph-mon[76349]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[100622]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:31:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:58 smithi184 ceph-mon[102844]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:00 smithi184 ceph-mon[100622]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:00 smithi184 ceph-mon[102844]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:00 smithi196 ceph-mon[76349]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:32:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:01.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:01.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:01.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:02 smithi184 ceph-mon[100622]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:02 smithi184 ceph-mon[102844]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:02 smithi196 ceph-mon[76349]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:04 smithi184 ceph-mon[100622]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:04 smithi184 ceph-mon[102844]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:04 smithi196 ceph-mon[76349]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:07.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:06 smithi184 ceph-mon[100622]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:06 smithi184 ceph-mon[102844]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:06 smithi196 ceph-mon[76349]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:07.783 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:07.783 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:08.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:08 smithi184 ceph-mon[100622]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:08 smithi184 ceph-mon[102844]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:08 smithi196 ceph-mon[76349]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:10 smithi184 ceph-mon[100622]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:10 smithi184 ceph-mon[102844]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:10 smithi196 ceph-mon[76349]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:32:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:11.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:11.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:11.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:12 smithi184 ceph-mon[100622]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:12 smithi184 ceph-mon[102844]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:12 smithi196 ceph-mon[76349]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:14 smithi184 ceph-mon[102844]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:14 smithi184 ceph-mon[100622]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:14 smithi196 ceph-mon[76349]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:16.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:16.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:16.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:16.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:16 smithi196 ceph-mon[76349]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:17.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[102844]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:16 smithi184 ceph-mon[100622]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:18.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:18 smithi196 ceph-mon[76349]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:19.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:18 smithi184 ceph-mon[100622]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:18 smithi184 ceph-mon[102844]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:20.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:20 smithi196 ceph-mon[76349]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:21.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:20 smithi184 ceph-mon[100622]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:20 smithi184 ceph-mon[102844]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:32:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:21.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:21.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:21.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:23.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:22 smithi184 ceph-mon[100622]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:22 smithi184 ceph-mon[102844]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:22 smithi196 ceph-mon[76349]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:24 smithi184 ceph-mon[100622]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:24 smithi184 ceph-mon[102844]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:24 smithi196 ceph-mon[76349]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:25.799 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 6h - - 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 6h - - 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 9s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 9s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:32:26.327 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 9s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 9s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 6h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 9s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 9s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 9s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 9s ago 6h 74.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:32:26.328 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:32:26.329 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:32:26.329 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:32:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:32:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:32:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:32:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:32:26.635 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:32:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:32:26.901 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:32:26.901 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[100622]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:26.901 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[100622]: from='client.71217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:26.902 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1352905269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:26.902 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[102844]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:26.902 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[102844]: from='client.71217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:26.902 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1352905269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:26 smithi196 ceph-mon[76349]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:26 smithi196 ceph-mon[76349]: from='client.71217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1352905269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:27.210 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:32:27.210 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:27.210 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:27.210 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:27.211 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:28.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:28.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[100622]: from='client.81188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[100622]: from='client.81194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[100622]: from='client.71241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.001 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1685492453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:28.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:28.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[102844]: from='client.81188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[102844]: from='client.81194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[102844]: from='client.71241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1685492453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:27 smithi196 ceph-mon[76349]: from='client.81188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:27 smithi196 ceph-mon[76349]: from='client.81194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:27 smithi196 ceph-mon[76349]: from='client.71241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1685492453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:28 smithi184 ceph-mon[100622]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:28 smithi184 ceph-mon[102844]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:28 smithi196 ceph-mon[76349]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:31.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:30 smithi184 ceph-mon[102844]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:31.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:30 smithi184 ceph-mon[100622]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:30 smithi196 ceph-mon[76349]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:32:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:31.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:31.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:31.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:32 smithi184 ceph-mon[100622]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:32 smithi184 ceph-mon[102844]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:32 smithi196 ceph-mon[76349]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:34 smithi184 ceph-mon[100622]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:34 smithi184 ceph-mon[102844]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:34 smithi196 ceph-mon[76349]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[100622]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[102844]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:36 smithi196 ceph-mon[76349]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:38.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:38 smithi196 ceph-mon[76349]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:39.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:38 smithi184 ceph-mon[100622]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:38 smithi184 ceph-mon[102844]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:41.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:40 smithi184 ceph-mon[100622]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:40 smithi184 ceph-mon[102844]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:40 smithi196 ceph-mon[76349]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:32:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:41.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:41.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:41.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:42 smithi184 ceph-mon[100622]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:42 smithi184 ceph-mon[102844]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:42 smithi196 ceph-mon[76349]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:45.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:44 smithi184 ceph-mon[100622]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:44 smithi184 ceph-mon[102844]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:44 smithi196 ceph-mon[76349]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:47.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:46 smithi184 ceph-mon[100622]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:46 smithi184 ceph-mon[102844]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:46 smithi196 ceph-mon[76349]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:49.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:48 smithi184 ceph-mon[100622]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:48 smithi184 ceph-mon[102844]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:48 smithi196 ceph-mon[76349]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:32:50.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:50 smithi184 ceph-mon[102844]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:50.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:50 smithi184 ceph-mon[100622]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:50 smithi196 ceph-mon[76349]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:32:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:32:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:51.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:51.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:32:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:32:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:32:51.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:32:53.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:52 smithi184 ceph-mon[102844]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:52 smithi184 ceph-mon[100622]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:52 smithi196 ceph-mon[76349]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:55.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:54 smithi184 ceph-mon[102844]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:54 smithi184 ceph-mon[100622]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:54 smithi196 ceph-mon[76349]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:56 smithi196 ceph-mon[76349]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[100622]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[102844]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:57.507 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 6h - - 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 2s ago 6h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 0s ago 6h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 2s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 0s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 2s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 2s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 0s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 2s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:32:58.042 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 2s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 0s ago 6h 18.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 2s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 2s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 2s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 2s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 0s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 0s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 0s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 0s ago 6h 75.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:32:58.043 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:32:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:32:58.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:32:58.356 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:32:58.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:32:58.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:32:58.356 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:32:58.625 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:32:58.626 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:32:58.937 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:32:58.937 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:58.938 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:58.938 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:58.938 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[100622]: from='client.71253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[100622]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[100622]: from='client.81224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/251727548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[102844]: from='client.71253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:58.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[102844]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:58.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[102844]: from='client.81224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:58.939 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/251727548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:58 smithi196 ceph-mon[76349]: from='client.71253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:58 smithi196 ceph-mon[76349]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:58 smithi196 ceph-mon[76349]: from='client.81224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/251727548' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:59 smithi196 ceph-mon[76349]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:59 smithi196 ceph-mon[76349]: from='client.71277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3895529110' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[102844]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[102844]: from='client.71277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3895529110' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[100622]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[100622]: from='client.71277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3895529110' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:00 smithi196 ceph-mon[76349]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:01.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:00 smithi184 ceph-mon[102844]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:00 smithi184 ceph-mon[100622]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:33:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:01.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:01.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:01.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:02 smithi196 ceph-mon[76349]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:03.215 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:02 smithi184 ceph-mon[100622]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:03.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:02 smithi184 ceph-mon[102844]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:04 smithi196 ceph-mon[76349]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:04 smithi184 ceph-mon[100622]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:04 smithi184 ceph-mon[102844]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:06 smithi196 ceph-mon[76349]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:06 smithi184 ceph-mon[100622]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:06 smithi184 ceph-mon[102844]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:08.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:08 smithi196 ceph-mon[76349]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:09.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:08 smithi184 ceph-mon[100622]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:09.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:08 smithi184 ceph-mon[102844]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:10 smithi196 ceph-mon[76349]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:11.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:10 smithi184 ceph-mon[100622]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:11.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:10 smithi184 ceph-mon[102844]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:33:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:11.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:11.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:11.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:12 smithi196 ceph-mon[76349]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:12 smithi184 ceph-mon[100622]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:12 smithi184 ceph-mon[102844]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:14 smithi196 ceph-mon[76349]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:15.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:14 smithi184 ceph-mon[100622]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:14 smithi184 ceph-mon[102844]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[102844]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[100622]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:16 smithi196 ceph-mon[76349]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:18 smithi196 ceph-mon[76349]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:18 smithi184 ceph-mon[100622]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:18 smithi184 ceph-mon[102844]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:20 smithi184 ceph-mon[100622]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:20 smithi184 ceph-mon[102844]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:20 smithi196 ceph-mon[76349]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:33:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:21.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:21.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:21.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:23.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:22 smithi184 ceph-mon[100622]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:22 smithi184 ceph-mon[102844]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:22 smithi196 ceph-mon[76349]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:24 smithi184 ceph-mon[100622]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:24 smithi184 ceph-mon[102844]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:24 smithi196 ceph-mon[76349]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:26 smithi184 ceph-mon[100622]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:26 smithi184 ceph-mon[102844]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:26 smithi196 ceph-mon[76349]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:28.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:28.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:28 smithi196 ceph-mon[76349]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:29.250 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:28 smithi184 ceph-mon[100622]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:29.251 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:28 smithi184 ceph-mon[102844]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:29.251 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:33:29.783 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 6h - - 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 13s ago 6h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 92.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 13s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 13s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 13s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 12s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 13s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 13s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 13s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:33:29.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 13s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 13s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 12s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 12s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 12s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 12s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 12s ago 6h 75.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:33:29.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 12s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:33:30.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:33:30.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:33:30.108 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:33:30.108 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:33:30.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:33:30.108 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:33:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:30.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:30.383 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:33:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:33:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:33:30.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:33:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:30.697 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:33:30.697 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:30.698 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:31.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: from='client.71289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:31.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: from='client.71301 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.226 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/778496957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1852174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: from='client.71289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: from='client.71301 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/778496957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:31.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1852174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: from='client.71289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: from='client.71301 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/778496957' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1852174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:32.275 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:32 smithi184 ceph-mon[100622]: from='client.71313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:32.275 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:33:32.276 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:31.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:32.276 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:31.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:32.276 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:31.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:32.276 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:32 smithi184 ceph-mon[102844]: from='client.71313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:32 smithi196 ceph-mon[76349]: from='client.71313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:33.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:33 smithi184 ceph-mon[100622]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:33.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:33 smithi184 ceph-mon[102844]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:33 smithi196 ceph-mon[76349]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:35 smithi184 ceph-mon[100622]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:35 smithi184 ceph-mon[102844]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:35 smithi196 ceph-mon[76349]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[100622]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[102844]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:37 smithi196 ceph-mon[76349]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:39 smithi184 ceph-mon[100622]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:39 smithi184 ceph-mon[102844]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:39 smithi196 ceph-mon[76349]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:41 smithi184 ceph-mon[100622]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:41 smithi184 ceph-mon[102844]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:41 smithi196 ceph-mon[76349]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:33:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:41.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:41.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:41.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:43 smithi184 ceph-mon[100622]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:43 smithi184 ceph-mon[102844]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:43 smithi196 ceph-mon[76349]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:45 smithi184 ceph-mon[100622]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:45 smithi184 ceph-mon[102844]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:45 smithi196 ceph-mon[76349]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:46.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:46 smithi184 ceph-mon[100622]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:46 smithi184 ceph-mon[102844]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:46 smithi196 ceph-mon[76349]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:47.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:48 smithi196 ceph-mon[76349]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:48 smithi184 ceph-mon[100622]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:48 smithi184 ceph-mon[102844]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:50 smithi184 ceph-mon[100622]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:50 smithi184 ceph-mon[102844]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:50 smithi196 ceph-mon[76349]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:51.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:51.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:33:52.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:33:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:33:51.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:33:52.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:33:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:33:52.879 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:52 smithi184 ceph-mon[102844]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:52.879 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:52 smithi184 ceph-mon[100622]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:52 smithi196 ceph-mon[76349]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:55.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:54 smithi184 ceph-mon[100622]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:54 smithi184 ceph-mon[102844]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:54 smithi196 ceph-mon[76349]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[100622]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[102844]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:56 smithi196 ceph-mon[76349]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:58.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:58.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:58 smithi196 ceph-mon[76349]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[100622]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[102844]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:33:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:00.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:00.999 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:34:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:00 smithi184 ceph-mon[100622]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:00 smithi184 ceph-mon[102844]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:00 smithi196 ceph-mon[76349]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 6h - - 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 3s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 3s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:34:01.530 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 6h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 3s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 3s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 3s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 3s ago 6h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:34:01.531 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 3s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:34:01.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:34:01.842 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:34:02.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:34:02.114 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:34:02.114 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-mon[100622]: from='client.71325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:02.114 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/801182605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:02.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-mon[102844]: from='client.71325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:02.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/801182605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:02.115 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:02] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:34:02.115 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:01.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:02.115 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:01.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:02.115 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:01.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:01 smithi196 ceph-mon[76349]: from='client.71325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/801182605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:02.426 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:34:02.427 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:02.427 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:02.427 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:02.427 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:02 smithi196 ceph-mon[76349]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:02 smithi196 ceph-mon[76349]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:02 smithi196 ceph-mon[76349]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1977687317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[100622]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[100622]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[100622]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1977687317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[102844]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[102844]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[102844]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1977687317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:04.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:03 smithi184 ceph-mon[100622]: from='client.71349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:03 smithi184 ceph-mon[102844]: from='client.71349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:03 smithi196 ceph-mon[76349]: from='client.71349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:04 smithi184 ceph-mon[100622]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:04 smithi184 ceph-mon[102844]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:04 smithi196 ceph-mon[76349]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:06 smithi184 ceph-mon[100622]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:06 smithi184 ceph-mon[102844]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:06 smithi196 ceph-mon[76349]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:08 smithi184 ceph-mon[100622]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:34:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:08 smithi184 ceph-mon[102844]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:34:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:08 smithi196 ceph-mon[76349]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:34:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:10 smithi184 ceph-mon[100622]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:34:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:10 smithi184 ceph-mon[102844]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:34:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:10 smithi196 ceph-mon[76349]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:34:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:12] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T02:34:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:11.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:11.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:11.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:12 smithi184 ceph-mon[100622]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:34:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:12 smithi184 ceph-mon[102844]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:34:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:12 smithi196 ceph-mon[76349]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:34:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:14 smithi184 ceph-mon[100622]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:34:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:14 smithi184 ceph-mon[102844]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:34:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:14 smithi196 ceph-mon[76349]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:34:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[100622]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:34:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[102844]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:16 smithi196 ceph-mon[76349]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:34:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[100622]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[102844]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:19 smithi196 ceph-mon[76349]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 363 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T02:34:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:21.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:21 smithi196 ceph-mon[76349]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 347 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:34:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:21 smithi184 ceph-mon[100622]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 347 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:34:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:21 smithi184 ceph-mon[102844]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 347 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-21T02:34:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:22 smithi184 ceph-mon[100622]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:34:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:22 smithi184 ceph-mon[102844]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:34:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:21.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:21.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:21.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:22.357 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:22 smithi196 ceph-mon[76349]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:34:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:24 smithi184 ceph-mon[100622]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:34:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:24 smithi184 ceph-mon[102844]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:34:25.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:24 smithi196 ceph-mon[76349]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:34:27.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:26 smithi184 ceph-mon[100622]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:34:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:26 smithi184 ceph-mon[102844]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:34:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:26 smithi196 ceph-mon[76349]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:34:28.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:28.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:28.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:29.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:28 smithi184 ceph-mon[100622]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:34:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:28 smithi184 ceph-mon[102844]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:34:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:28 smithi196 ceph-mon[76349]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 351 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:34:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:30 smithi184 ceph-mon[102844]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:30 smithi184 ceph-mon[100622]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:30 smithi196 ceph-mon[76349]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:34:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:31.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:31.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:31.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:32.731 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:34:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:32 smithi184 ceph-mon[100622]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:34:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:32 smithi184 ceph-mon[102844]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:34:33.287 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:34:33.287 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 6h - - 2024-09-21T02:34:33.287 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 16s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 16s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 16s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 15s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 16s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 16s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 16s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 16s ago 6h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:34:33.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 16s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 15s ago 6h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 15s ago 6h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 15s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 15s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 15s ago 6h 76.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:34:33.289 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 15s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:34:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:32 smithi196 ceph-mon[76349]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:34:33.610 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:34:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:34:33.611 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:34:33.883 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:34:33.884 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:34:33.884 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:34:33.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[100622]: from='client.71361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[100622]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/328933954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[102844]: from='client.71361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[102844]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/328933954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:34.225 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:34:34.225 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:34.225 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:34.225 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:34.225 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:33 smithi196 ceph-mon[76349]: from='client.71361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:33 smithi196 ceph-mon[76349]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/328933954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:35.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:34 smithi196 ceph-mon[76349]: from='client.81338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:34 smithi196 ceph-mon[76349]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:34:35.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:34 smithi196 ceph-mon[76349]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4290148954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[100622]: from='client.81338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[100622]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[100622]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4290148954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[102844]: from='client.81338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[102844]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 355 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[102844]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4290148954' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[100622]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[102844]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:36 smithi196 ceph-mon[76349]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:34:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:38.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[100622]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[102844]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:38 smithi196 ceph-mon[76349]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:34:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:41.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:40 smithi184 ceph-mon[100622]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:40 smithi184 ceph-mon[102844]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:40 smithi196 ceph-mon[76349]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:34:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:34:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:41.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:41.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:41.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:43.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:42 smithi196 ceph-mon[76349]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:34:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:42 smithi184 ceph-mon[100622]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:34:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:42 smithi184 ceph-mon[102844]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T02:34:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:44 smithi184 ceph-mon[100622]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:34:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:44 smithi184 ceph-mon[102844]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:34:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:44 smithi196 ceph-mon[76349]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:34:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:47.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:46 smithi184 ceph-mon[100622]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:34:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:46 smithi184 ceph-mon[102844]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:34:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:46 smithi196 ceph-mon[76349]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:34:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:48 smithi184 ceph-mon[100622]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:34:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:48 smithi184 ceph-mon[102844]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:34:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:48 smithi196 ceph-mon[76349]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:34:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:50 smithi184 ceph-mon[100622]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:34:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:50 smithi184 ceph-mon[102844]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:34:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:50 smithi196 ceph-mon[76349]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:34:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:34:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:34:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:51.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:51.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:34:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:34:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:34:51.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:34:53.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:52 smithi184 ceph-mon[100622]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:34:53.294 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:52 smithi184 ceph-mon[102844]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:34:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:52 smithi196 ceph-mon[76349]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:34:55.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:54 smithi184 ceph-mon[102844]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:55.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:54 smithi184 ceph-mon[100622]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:54 smithi196 ceph-mon[76349]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[100622]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[102844]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:56 smithi196 ceph-mon[76349]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:58.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:58.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:58 smithi196 ceph-mon[76349]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[100622]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[102844]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:34:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:34:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:34:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:34:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:00 smithi184 ceph-mon[100622]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:00 smithi184 ceph-mon[102844]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:00 smithi196 ceph-mon[76349]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:35:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:01.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:01.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:01.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:03.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:02 smithi196 ceph-mon[76349]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:02 smithi184 ceph-mon[100622]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:02 smithi184 ceph-mon[102844]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:04.519 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 6h - - 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 6h - - 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 6s ago 6h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 6s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:35:05.045 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 6s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 8s ago 6h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 6s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 6s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 6s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:35:05.046 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 6s ago 6h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:35:05.047 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 6s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:35:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:04 smithi196 ceph-mon[76349]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:05.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:04 smithi184 ceph-mon[102844]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:04 smithi184 ceph-mon[100622]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:05.357 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:05.358 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:35:05.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:35:05.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:35:05.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:35:05.628 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:35:05.628 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:35:05.628 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:35:05.628 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:35:05.628 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:35:05.629 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:35:05.937 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:35:05.938 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:05.938 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:05.938 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:05.938 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:06.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[102844]: from='client.71397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[102844]: from='client.81368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[102844]: from='client.81374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2919634983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/391270731' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[100622]: from='client.71397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[100622]: from='client.81368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[100622]: from='client.81374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2919634983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:06.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/391270731' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:05 smithi196 ceph-mon[76349]: from='client.71397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:05 smithi196 ceph-mon[76349]: from='client.81368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:05 smithi196 ceph-mon[76349]: from='client.81374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2919634983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/391270731' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:06 smithi184 ceph-mon[100622]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:06 smithi184 ceph-mon[100622]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:06 smithi184 ceph-mon[102844]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:06 smithi184 ceph-mon[102844]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:06 smithi196 ceph-mon[76349]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:06 smithi196 ceph-mon[76349]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:08 smithi184 ceph-mon[100622]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:08 smithi184 ceph-mon[102844]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:08 smithi196 ceph-mon[76349]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:10 smithi184 ceph-mon[100622]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:10 smithi184 ceph-mon[102844]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:10 smithi196 ceph-mon[76349]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:35:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:11.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:11.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:11.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:13.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:12 smithi184 ceph-mon[100622]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:12 smithi184 ceph-mon[102844]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:12 smithi196 ceph-mon[76349]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:15.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:14 smithi184 ceph-mon[102844]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:15.032 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:15.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:14 smithi184 ceph-mon[100622]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:15.032 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:14 smithi196 ceph-mon[76349]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[102844]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[100622]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:16 smithi196 ceph-mon[76349]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[100622]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[102844]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:19 smithi196 ceph-mon[76349]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:21 smithi184 ceph-mon[100622]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:21 smithi184 ceph-mon[102844]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:21 smithi196 ceph-mon[76349]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:35:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:21.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:21.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:21.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:23.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:23 smithi196 ceph-mon[76349]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:23 smithi184 ceph-mon[100622]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:23 smithi184 ceph-mon[102844]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:25 smithi196 ceph-mon[76349]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:25 smithi184 ceph-mon[100622]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:25 smithi184 ceph-mon[102844]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:27.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:27 smithi184 ceph-mon[100622]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:27 smithi184 ceph-mon[102844]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:27 smithi196 ceph-mon[76349]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:28.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:28.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:29 smithi184 ceph-mon[100622]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:29 smithi184 ceph-mon[102844]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:29 smithi196 ceph-mon[76349]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:30.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:31 smithi184 ceph-mon[100622]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:31 smithi184 ceph-mon[102844]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:31 smithi196 ceph-mon[76349]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:32.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:35:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:31.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:32.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:31.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:32.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:31.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:33 smithi184 ceph-mon[100622]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:33 smithi184 ceph-mon[102844]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:33 smithi196 ceph-mon[76349]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:35 smithi184 ceph-mon[100622]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:35 smithi184 ceph-mon[102844]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:35 smithi196 ceph-mon[76349]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:36.229 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:35:36.764 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:35:36.764 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 20s ago 6h - - 2024-09-21T02:35:36.764 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T02:35:36.764 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 20s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 20s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 20s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 20s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 20s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 20s ago 6h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 20s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 20s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 20s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 20s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:35:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 6h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 20s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 20s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:35:36.766 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:35:37.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:35:37.084 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:35:37.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[102844]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[100622]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:35:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:35:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:35:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:37 smithi196 ceph-mon[76349]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:37.660 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:35:37.661 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:37.661 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.71433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.81404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.71445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3207631815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3720800601' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.71433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.81404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.71445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3207631815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3720800601' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.71433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.81404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.71445 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3207631815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3720800601' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[100622]: from='client.71457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[100622]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[102844]: from='client.71457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[102844]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:39 smithi196 ceph-mon[76349]: from='client.71457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:39 smithi196 ceph-mon[76349]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:41 smithi196 ceph-mon[76349]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:41.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:41 smithi184 ceph-mon[100622]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:41 smithi184 ceph-mon[102844]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:35:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:41.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:41.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:41.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:43.375 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:43 smithi196 ceph-mon[76349]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:43.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:43 smithi184 ceph-mon[100622]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:43 smithi184 ceph-mon[102844]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:45.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:45 smithi196 ceph-mon[76349]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:45.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:45 smithi184 ceph-mon[100622]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:45 smithi184 ceph-mon[102844]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:47 smithi196 ceph-mon[76349]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:47 smithi184 ceph-mon[100622]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:47 smithi184 ceph-mon[102844]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:48.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:49 smithi196 ceph-mon[76349]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:49.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:49 smithi184 ceph-mon[100622]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:49 smithi184 ceph-mon[102844]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:50 smithi196 ceph-mon[76349]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:50.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:50 smithi184 ceph-mon[100622]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:50 smithi184 ceph-mon[102844]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:35:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:35:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:35:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:51.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:51.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:35:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:35:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:35:51.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:35:53.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:52 smithi184 ceph-mon[100622]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:53.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:52 smithi184 ceph-mon[102844]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:52 smithi196 ceph-mon[76349]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:55.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:54 smithi184 ceph-mon[100622]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:55.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:54 smithi184 ceph-mon[102844]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:54 smithi196 ceph-mon[76349]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:56.998 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:56 smithi184 ceph-mon[100622]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:56.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:56 smithi184 ceph-mon[102844]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:56 smithi196 ceph-mon[76349]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[102844]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[100622]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:35:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:58 smithi196 ceph-mon[76349]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:35:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:35:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:00.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:01 smithi184 ceph-mon[100622]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:01 smithi184 ceph-mon[102844]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:01 smithi196 ceph-mon[76349]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:02.199 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:36:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:01.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:02.199 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:01.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:02.200 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:01.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:03 smithi184 ceph-mon[100622]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:03 smithi184 ceph-mon[102844]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:03 smithi196 ceph-mon[76349]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:05 smithi184 ceph-mon[100622]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:05 smithi184 ceph-mon[102844]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:05.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:05 smithi196 ceph-mon[76349]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:07 smithi184 ceph-mon[100622]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:07 smithi184 ceph-mon[102844]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:07 smithi196 ceph-mon[76349]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:07.959 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:36:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 6h - - 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 6h - - 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 11s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 9s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:36:08.489 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 9s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 9s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 11s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 9s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 9s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 9s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:36:08.490 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 9s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 9s ago 6h 76.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:36:08.491 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:36:08.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:36:08.801 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:36:09.095 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:36:09.096 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[100622]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[100622]: from='client.81434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3143503864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[102844]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[102844]: from='client.81434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:09.097 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3143503864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:09 smithi196 ceph-mon[76349]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:09 smithi196 ceph-mon[76349]: from='client.81434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3143503864' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:09.408 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:36:09.408 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:09.408 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:10.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[102844]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[102844]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2941751940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[100622]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[100622]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2941751940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:10 smithi196 ceph-mon[76349]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:10 smithi196 ceph-mon[76349]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2941751940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-mon[102844]: from='client.81458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-mon[102844]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-mon[100622]: from='client.81458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-mon[100622]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:11 smithi196 ceph-mon[76349]: from='client.81458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:11 smithi196 ceph-mon[76349]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:36:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:11.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:11.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:11.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:13 smithi184 ceph-mon[100622]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:13 smithi184 ceph-mon[102844]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:13 smithi196 ceph-mon[76349]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:15 smithi184 ceph-mon[102844]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:15 smithi184 ceph-mon[100622]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:15 smithi196 ceph-mon[76349]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:17 smithi184 ceph-mon[100622]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:17 smithi184 ceph-mon[102844]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:17 smithi196 ceph-mon[76349]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:19 smithi196 ceph-mon[76349]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[102844]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[100622]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:21.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:21 smithi184 ceph-mon[100622]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:21 smithi184 ceph-mon[102844]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:21 smithi196 ceph-mon[76349]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:36:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:21.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:21.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:21.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:23 smithi196 ceph-mon[76349]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:23 smithi184 ceph-mon[100622]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:23 smithi184 ceph-mon[102844]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:25 smithi196 ceph-mon[76349]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:25.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:25 smithi184 ceph-mon[100622]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:25 smithi184 ceph-mon[102844]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:27.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:27 smithi196 ceph-mon[76349]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:27.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:27 smithi184 ceph-mon[100622]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:27 smithi184 ceph-mon[102844]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:28.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:29 smithi196 ceph-mon[76349]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:29 smithi184 ceph-mon[100622]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:29 smithi184 ceph-mon[102844]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:30 smithi196 ceph-mon[76349]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:30 smithi184 ceph-mon[100622]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:30 smithi184 ceph-mon[102844]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:32.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:36:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:31.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:31.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:31.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:32 smithi184 ceph-mon[100622]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:32 smithi184 ceph-mon[102844]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:32 smithi196 ceph-mon[76349]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:34 smithi184 ceph-mon[100622]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:34 smithi184 ceph-mon[102844]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:34 smithi196 ceph-mon[76349]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:36 smithi184 ceph-mon[102844]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:36 smithi184 ceph-mon[100622]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:36 smithi196 ceph-mon[76349]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:37.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:38.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:39.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:38 smithi184 ceph-mon[100622]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:39.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:38 smithi184 ceph-mon[102844]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:38 smithi196 ceph-mon[76349]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:39.704 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:36:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 6h - - 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 2s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 2s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:36:40.233 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 2s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 2s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 2s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 2s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 2s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 2s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 2s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:36:40.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:36:40.235 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:36:40.235 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:36:40.235 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:36:40.235 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:36:40.235 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:36:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:36:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:36:40.542 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:36:40.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:36:40.812 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:36:40.812 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:36:40.812 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:36:40.812 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:36:40.813 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:36:41.121 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:36:41.121 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:41.121 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:41.121 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:41.121 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:41.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[100622]: from='client.81470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.121 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[100622]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[100622]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3472057294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[102844]: from='client.81470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[102844]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[102844]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3472057294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:40 smithi196 ceph-mon[76349]: from='client.81470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:40 smithi196 ceph-mon[76349]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:40 smithi196 ceph-mon[76349]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3472057294' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[100622]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[100622]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1228100515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:36:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[102844]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[102844]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1228100515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:41.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:41.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:41.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:41 smithi196 ceph-mon[76349]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:41 smithi196 ceph-mon[76349]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1228100515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:43.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:42 smithi184 ceph-mon[100622]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:42 smithi184 ceph-mon[102844]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:42 smithi196 ceph-mon[76349]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:44 smithi184 ceph-mon[100622]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:44 smithi184 ceph-mon[102844]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:44 smithi196 ceph-mon[76349]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:46 smithi196 ceph-mon[76349]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:46 smithi184 ceph-mon[100622]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:46 smithi184 ceph-mon[102844]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:48.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:48.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:49.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:48 smithi184 ceph-mon[102844]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:48 smithi184 ceph-mon[100622]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:48 smithi196 ceph-mon[76349]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:50 smithi184 ceph-mon[102844]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:50 smithi184 ceph-mon[100622]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:50 smithi196 ceph-mon[76349]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:36:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:36:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:36:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:51.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:51.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:36:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:36:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:36:51.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:36:53.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:52 smithi184 ceph-mon[102844]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:52 smithi184 ceph-mon[100622]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:52 smithi196 ceph-mon[76349]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:54 smithi184 ceph-mon[102844]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:54 smithi184 ceph-mon[100622]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:54 smithi196 ceph-mon[76349]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:57.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:56 smithi184 ceph-mon[100622]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:57.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:56 smithi184 ceph-mon[102844]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:56 smithi196 ceph-mon[76349]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:58.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:58.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.279 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.279 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:58.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:58.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:58.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:36:59.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:36:58 smithi184 ceph-mon[100622]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:59.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:36:58 smithi184 ceph-mon[102844]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:36:58 smithi196 ceph-mon[76349]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:01 smithi196 ceph-mon[76349]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:01 smithi184 ceph-mon[100622]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:01 smithi184 ceph-mon[102844]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:02 smithi184 ceph-mon[100622]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:02 smithi184 ceph-mon[102844]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:37:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:01.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:01.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:01.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:02 smithi196 ceph-mon[76349]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:05.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:04 smithi184 ceph-mon[100622]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:04 smithi184 ceph-mon[102844]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:04 smithi196 ceph-mon[76349]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:07.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:06 smithi184 ceph-mon[102844]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:06 smithi184 ceph-mon[100622]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:06 smithi196 ceph-mon[76349]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:07.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:08 smithi184 ceph-mon[100622]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:08 smithi184 ceph-mon[102844]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:08 smithi196 ceph-mon[76349]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:10 smithi184 ceph-mon[100622]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:10 smithi184 ceph-mon[102844]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:10 smithi196 ceph-mon[76349]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:11.414 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:37:11.940 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 6h - - 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 6h - - 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 14s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 12s ago 6h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 12s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 12s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:37:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 6h 389M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 14s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 12s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 12s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 12s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 12s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 12s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 12s ago 6h 71.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:37:11.942 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:37:11.943 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:37:11.943 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 12s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:37:12.251 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:37:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:37:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:37:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:37:12.252 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:37:12.253 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:37:12.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:11.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:12.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:11.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:12.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:11.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:12.520 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:37:12.520 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:37:12.521 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:37:12.833 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:37:12.833 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:12.833 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:12.833 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:12.833 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:13.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[100622]: from='client.71541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[100622]: from='client.81512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[100622]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[100622]: from='client.71553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2104471878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[102844]: from='client.71541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[102844]: from='client.81512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[102844]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[102844]: from='client.71553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2104471878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:12 smithi196 ceph-mon[76349]: from='client.71541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:12 smithi196 ceph-mon[76349]: from='client.81512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:12 smithi196 ceph-mon[76349]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:12 smithi196 ceph-mon[76349]: from='client.71553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:13.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2104471878' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:14.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:13 smithi184 ceph-mon[100622]: from='client.81530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/950292680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:13 smithi184 ceph-mon[102844]: from='client.81530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/950292680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:13 smithi196 ceph-mon[76349]: from='client.81530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/950292680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:15.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:14 smithi184 ceph-mon[100622]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:14 smithi184 ceph-mon[102844]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:14 smithi196 ceph-mon[76349]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:16 smithi184 ceph-mon[100622]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:16 smithi184 ceph-mon[102844]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:16 smithi196 ceph-mon[76349]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:19.024 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:18 smithi184 ceph-mon[100622]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:19.024 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:18 smithi184 ceph-mon[102844]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:18 smithi196 ceph-mon[76349]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[100622]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:20 smithi184 ceph-mon[102844]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:20 smithi196 ceph-mon[76349]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:37:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:21.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:21.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:21.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:22 smithi184 ceph-mon[100622]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:22 smithi184 ceph-mon[102844]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:22 smithi196 ceph-mon[76349]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:24 smithi184 ceph-mon[100622]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:24 smithi184 ceph-mon[102844]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:24 smithi196 ceph-mon[76349]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:27.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:26 smithi196 ceph-mon[76349]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:27.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:26 smithi184 ceph-mon[102844]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:26 smithi184 ceph-mon[100622]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:28.101 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:28 smithi184 ceph-mon[102844]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:28 smithi184 ceph-mon[100622]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:28 smithi196 ceph-mon[76349]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:30.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:30 smithi184 ceph-mon[100622]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:30 smithi184 ceph-mon[102844]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:30 smithi196 ceph-mon[76349]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:37:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:31.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:31.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:31.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:33.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:32 smithi184 ceph-mon[100622]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:33.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:32 smithi184 ceph-mon[102844]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:32 smithi196 ceph-mon[76349]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:35.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:34 smithi184 ceph-mon[100622]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:35.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:34 smithi184 ceph-mon[102844]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:34 smithi196 ceph-mon[76349]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:36 smithi196 ceph-mon[76349]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:36 smithi184 ceph-mon[100622]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:36 smithi184 ceph-mon[102844]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:38 smithi196 ceph-mon[76349]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:39.213 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:38 smithi184 ceph-mon[100622]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:39.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:38 smithi184 ceph-mon[102844]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:40 smithi196 ceph-mon[76349]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[100622]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:40.670 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:40 smithi184 ceph-mon[102844]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:37:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:41.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:41.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:41.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:42 smithi184 ceph-mon[100622]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:42 smithi184 ceph-mon[102844]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:43.129 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:37:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:42 smithi196 ceph-mon[76349]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:43.655 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:37:43.655 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 6h - - 2024-09-21T02:37:43.655 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T02:37:43.655 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:37:43.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 6h 74.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:37:43.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:37:43.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:37:43.968 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:37:43.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:37:43.969 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:37:44.238 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:37:44.239 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:37:44.545 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:37:44.545 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:44.545 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:44.545 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:44.545 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='client.71577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='client.71583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3171001974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/711116383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='client.71577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='client.71583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3171001974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/711116383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='client.71577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='client.71583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3171001974' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/711116383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:46.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:45 smithi184 ceph-mon[100622]: from='client.71601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:45 smithi184 ceph-mon[102844]: from='client.71601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:45 smithi196 ceph-mon[76349]: from='client.71601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:46 smithi184 ceph-mon[100622]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:46 smithi184 ceph-mon[102844]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:46 smithi196 ceph-mon[76349]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:47.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:48.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:49.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:48 smithi196 ceph-mon[76349]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:48 smithi184 ceph-mon[100622]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:48 smithi184 ceph-mon[102844]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:50 smithi196 ceph-mon[76349]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:50 smithi184 ceph-mon[100622]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:50 smithi184 ceph-mon[102844]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:37:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:37:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:37:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:51.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:51.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:37:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:37:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:37:51.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:37:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:52 smithi196 ceph-mon[76349]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:52 smithi184 ceph-mon[100622]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:52 smithi184 ceph-mon[102844]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:54 smithi196 ceph-mon[76349]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:54 smithi184 ceph-mon[100622]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:54 smithi184 ceph-mon[102844]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:56 smithi196 ceph-mon[76349]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:56 smithi184 ceph-mon[100622]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:56 smithi184 ceph-mon[102844]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:37:59.106 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:58 smithi184 ceph-mon[100622]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:59.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:58 smithi184 ceph-mon[102844]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:59.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:58 smithi196 ceph-mon[76349]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:00.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.095 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.095 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:37:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:37:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:01.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:00 smithi184 ceph-mon[100622]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:01.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:00 smithi184 ceph-mon[102844]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:00 smithi196 ceph-mon[76349]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:02.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:38:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:01.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:01.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:02.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:01.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:02 smithi196 ceph-mon[76349]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:02 smithi184 ceph-mon[100622]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:02 smithi184 ceph-mon[102844]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:04 smithi196 ceph-mon[76349]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:04 smithi184 ceph-mon[100622]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:04 smithi184 ceph-mon[102844]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:06 smithi196 ceph-mon[76349]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:07.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:06 smithi184 ceph-mon[100622]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:06 smithi184 ceph-mon[102844]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:08.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:08 smithi196 ceph-mon[76349]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:08 smithi184 ceph-mon[100622]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:08 smithi184 ceph-mon[102844]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:10 smithi196 ceph-mon[76349]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:10 smithi184 ceph-mon[102844]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:10 smithi184 ceph-mon[100622]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:38:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:11.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:11.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:11.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:12 smithi196 ceph-mon[76349]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:12 smithi184 ceph-mon[100622]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:12 smithi184 ceph-mon[102844]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:38:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:38:13.81272748Z level=info msg="Completed cleanup jobs" duration=12.849297ms 2024-09-21T02:38:14.839 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:38:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:14 smithi196 ceph-mon[76349]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:14 smithi184 ceph-mon[102844]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:14 smithi184 ceph-mon[100622]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:15.368 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:38:15.368 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 6h - - 2024-09-21T02:38:15.368 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 6h - - 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 17s ago 6h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 15s ago 6h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 17s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 15s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 17s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 17s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 15s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 17s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 17s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 15s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 17s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 17s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 17s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:38:15.369 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 15s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 15s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 15s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 15s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 15s ago 6h 74.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:38:15.370 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 15s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:38:15.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:38:15.679 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:38:15.956 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:38:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:38:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:38:15.957 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:38:15.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:15 smithi184 ceph-mon[100622]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:15.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/539180719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:15.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:15 smithi184 ceph-mon[102844]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:15.958 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/539180719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:15 smithi196 ceph-mon[76349]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/539180719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:16.265 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:38:16.265 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:16.265 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:16.265 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:16.265 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:16 smithi196 ceph-mon[76349]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:16 smithi196 ceph-mon[76349]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:16 smithi196 ceph-mon[76349]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:16 smithi196 ceph-mon[76349]: from='client.81602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1688053341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[100622]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[100622]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[100622]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[100622]: from='client.81602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1688053341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[102844]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[102844]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[102844]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[102844]: from='client.81602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1688053341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[100622]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[102844]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:18 smithi196 ceph-mon[76349]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[100622]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:20 smithi184 ceph-mon[102844]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:20 smithi196 ceph-mon[76349]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:21.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:21.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:21.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:22.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:38:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:22 smithi184 ceph-mon[100622]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:22 smithi184 ceph-mon[102844]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:22 smithi196 ceph-mon[76349]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:25.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:24 smithi184 ceph-mon[100622]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:24 smithi184 ceph-mon[102844]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:24 smithi196 ceph-mon[76349]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:26 smithi184 ceph-mon[100622]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:26 smithi184 ceph-mon[102844]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:26 smithi196 ceph-mon[76349]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:28.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:29.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:28 smithi196 ceph-mon[76349]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:28 smithi184 ceph-mon[100622]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:28 smithi184 ceph-mon[102844]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:30 smithi184 ceph-mon[100622]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:30 smithi184 ceph-mon[102844]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:30 smithi196 ceph-mon[76349]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:32.214 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:38:32.214 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:31.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:32.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:31.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:32.215 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:31.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:33.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:32 smithi184 ceph-mon[102844]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:32 smithi184 ceph-mon[100622]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:32 smithi196 ceph-mon[76349]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:34 smithi184 ceph-mon[100622]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:34 smithi184 ceph-mon[102844]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:34 smithi196 ceph-mon[76349]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:36 smithi184 ceph-mon[100622]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:36 smithi184 ceph-mon[102844]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:36 smithi196 ceph-mon[76349]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[100622]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[102844]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:39 smithi196 ceph-mon[76349]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[100622]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:40 smithi184 ceph-mon[102844]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:41.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:41.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:40 smithi196 ceph-mon[76349]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:38:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:41.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:41.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:41.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:43.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:42 smithi184 ceph-mon[100622]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:42 smithi184 ceph-mon[102844]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:42 smithi196 ceph-mon[76349]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:44 smithi196 ceph-mon[76349]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:44 smithi184 ceph-mon[100622]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:44 smithi184 ceph-mon[102844]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:46.560 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:38:47.084 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:46 smithi184 ceph-mon[100622]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:47.084 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:46 smithi184 ceph-mon[102844]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:47.084 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:38:47.084 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 6h - - 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 9s ago 6h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 7s ago 6h 94.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 9s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 7s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 9s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 9s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 9s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 7s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 9s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 9s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 9s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:38:47.085 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 7s ago 6h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:38:47.086 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:38:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:46 smithi196 ceph-mon[76349]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:38:47.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:38:47.399 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:38:47.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:38:47.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:38:47.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:38:47.670 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:38:47.670 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:38:47.670 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:38:47.670 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:38:47.670 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:38:47.671 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:38:47.987 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:38:47.988 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:47.988 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[100622]: from='client.81614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[100622]: from='client.71655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[100622]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/203413162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:47.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:47.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[102844]: from='client.81614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[102844]: from='client.71655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[102844]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:47.989 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/203413162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:47 smithi196 ceph-mon[76349]: from='client.81614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:47 smithi196 ceph-mon[76349]: from='client.71655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:47 smithi196 ceph-mon[76349]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/203413162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:48 smithi196 ceph-mon[76349]: from='client.71673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:48 smithi196 ceph-mon[76349]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/363137143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[102844]: from='client.71673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[102844]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/363137143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[100622]: from='client.71673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[100622]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/363137143' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:51.093 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:50 smithi196 ceph-mon[76349]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:50 smithi184 ceph-mon[100622]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:50 smithi184 ceph-mon[102844]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:38:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:38:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:38:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:51.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:51.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:38:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:38:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:38:51.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:38:53.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:52 smithi196 ceph-mon[76349]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:53.149 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:52 smithi184 ceph-mon[100622]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:53.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:52 smithi184 ceph-mon[102844]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:55.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:54 smithi196 ceph-mon[76349]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:54 smithi184 ceph-mon[100622]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:54 smithi184 ceph-mon[102844]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:56 smithi196 ceph-mon[76349]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:56 smithi184 ceph-mon[100622]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:56 smithi184 ceph-mon[102844]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:58.185 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:58.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:58 smithi196 ceph-mon[76349]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[102844]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[100622]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:38:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:38:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:38:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:38:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:01.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[100622]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:01.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.108 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.108 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[102844]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:01.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:00 smithi196 ceph-mon[76349]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:01.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:01.186 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:02.244 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:39:02.244 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:01.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:02.244 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:01.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:02.244 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:01.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:02 smithi184 ceph-mon[100622]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:02 smithi184 ceph-mon[102844]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:02 smithi196 ceph-mon[76349]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:04 smithi196 ceph-mon[76349]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:05.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:04 smithi184 ceph-mon[100622]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:04 smithi184 ceph-mon[102844]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:06 smithi196 ceph-mon[76349]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:06 smithi184 ceph-mon[100622]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:06 smithi184 ceph-mon[102844]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:08 smithi196 ceph-mon[76349]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:08 smithi184 ceph-mon[100622]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:08 smithi184 ceph-mon[102844]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:10 smithi196 ceph-mon[76349]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:10 smithi184 ceph-mon[100622]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:10 smithi184 ceph-mon[102844]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:12.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:39:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:11.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:11.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:11.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:13.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:12 smithi196 ceph-mon[76349]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:12 smithi184 ceph-mon[100622]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:12 smithi184 ceph-mon[102844]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:14 smithi196 ceph-mon[76349]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:15.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:14 smithi184 ceph-mon[102844]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:14 smithi184 ceph-mon[100622]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:17.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:16 smithi196 ceph-mon[76349]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:16 smithi184 ceph-mon[100622]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:16 smithi184 ceph-mon[102844]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:18.288 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:18.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:18.289 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 6h - - 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 6h - - 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 0s ago 6h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 18s ago 6h 96.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 0s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:39:18.817 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 0s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 0s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 0s ago 6h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 0s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 18s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 6h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:39:18.818 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:39:18.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:39:18.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:39:18.819 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:39:19.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:39:19.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:39:19.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[100622]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[102844]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:18 smithi196 ceph-mon[76349]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:39:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:39:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:39:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:39:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:39:19.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:39:19.703 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:39:19.703 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:19.703 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:19.703 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:19.703 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[100622]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[100622]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[100622]: from='client.81662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4010731349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4179256869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[102844]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[102844]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[102844]: from='client.81662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4010731349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4179256869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:19 smithi196 ceph-mon[76349]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:19 smithi196 ceph-mon[76349]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:19 smithi196 ceph-mon[76349]: from='client.81662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4010731349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4179256869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[100622]: from='client.71709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[100622]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[102844]: from='client.71709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[102844]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:20 smithi196 ceph-mon[76349]: from='client.71709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:20 smithi196 ceph-mon[76349]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:39:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:21.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:21.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:21.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:23.122 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:22 smithi184 ceph-mon[100622]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:23.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:22 smithi184 ceph-mon[102844]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:22 smithi196 ceph-mon[76349]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:25.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:24 smithi184 ceph-mon[100622]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:24 smithi184 ceph-mon[102844]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:24 smithi196 ceph-mon[76349]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:26 smithi184 ceph-mon[100622]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:26 smithi184 ceph-mon[102844]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:26 smithi196 ceph-mon[76349]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:28.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:28 smithi196 ceph-mon[76349]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:28 smithi184 ceph-mon[100622]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:28 smithi184 ceph-mon[102844]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:31 smithi184 ceph-mon[100622]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:31 smithi184 ceph-mon[102844]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:31.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:31 smithi196 ceph-mon[76349]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:39:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:31.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:31.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:31.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:33.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:33 smithi196 ceph-mon[76349]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:33 smithi184 ceph-mon[100622]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:33 smithi184 ceph-mon[102844]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:35.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:35 smithi184 ceph-mon[100622]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:35 smithi184 ceph-mon[102844]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:35 smithi196 ceph-mon[76349]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:37.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:37 smithi184 ceph-mon[100622]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:37 smithi184 ceph-mon[102844]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:37 smithi196 ceph-mon[76349]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:39.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[100622]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[102844]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:39 smithi196 ceph-mon[76349]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[100622]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[102844]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:41 smithi196 ceph-mon[76349]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:39:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:41.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:41.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:41.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:43.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:43 smithi184 ceph-mon[100622]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:43.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:43 smithi184 ceph-mon[102844]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:43 smithi196 ceph-mon[76349]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:45 smithi184 ceph-mon[100622]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:45 smithi184 ceph-mon[102844]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:45 smithi196 ceph-mon[76349]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:47 smithi184 ceph-mon[100622]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:47 smithi184 ceph-mon[102844]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:47 smithi196 ceph-mon[76349]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:49 smithi184 ceph-mon[100622]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:49 smithi184 ceph-mon[102844]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:49 smithi196 ceph-mon[76349]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:50.003 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 6h - - 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 6h - - 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 10s ago 6h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 10s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 12s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 10s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 12s ago 6h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:39:50.532 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 10s ago 6h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 12s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 12s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 6h 77.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:39:50.533 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:39:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:39:50.843 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:39:51.114 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:39:51.115 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[100622]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[100622]: from='client.71721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2212625746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[102844]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[102844]: from='client.71721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:51.116 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2212625746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:51 smithi196 ceph-mon[76349]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:51 smithi196 ceph-mon[76349]: from='client.71721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2212625746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:51.426 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:39:51.426 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:51.427 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:51.427 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:51.427 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:39:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[100622]: from='client.81692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[100622]: from='client.71733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4257070051' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[102844]: from='client.81692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[102844]: from='client.71733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4257070051' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:51.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:51.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:39:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:39:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:39:51.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:39:52.341 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:52 smithi196 ceph-mon[76349]: from='client.81692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:52 smithi196 ceph-mon[76349]: from='client.71733 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:52.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4257070051' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:53 smithi196 ceph-mon[76349]: from='client.71745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:53 smithi196 ceph-mon[76349]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:53.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:53 smithi184 ceph-mon[100622]: from='client.71745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:53 smithi184 ceph-mon[100622]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:53 smithi184 ceph-mon[102844]: from='client.71745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:53 smithi184 ceph-mon[102844]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:55 smithi196 ceph-mon[76349]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:55 smithi184 ceph-mon[100622]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:55 smithi184 ceph-mon[102844]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:57 smithi196 ceph-mon[76349]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:57 smithi184 ceph-mon[100622]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:57 smithi184 ceph-mon[102844]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:59 smithi196 ceph-mon[76349]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:39:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[100622]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[102844]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:39:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:39:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:39:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:01 smithi196 ceph-mon[76349]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[100622]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[102844]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:40:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:01.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:01.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:01.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:03.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:03 smithi184 ceph-mon[100622]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:03.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:03 smithi184 ceph-mon[102844]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:03 smithi196 ceph-mon[76349]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:05 smithi196 ceph-mon[76349]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:05.400 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:05 smithi184 ceph-mon[100622]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:05.400 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:05 smithi184 ceph-mon[102844]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:07 smithi196 ceph-mon[76349]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:07.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:07 smithi184 ceph-mon[100622]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:07 smithi184 ceph-mon[102844]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:09 smithi196 ceph-mon[76349]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:09 smithi184 ceph-mon[100622]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:09 smithi184 ceph-mon[102844]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:11 smithi196 ceph-mon[76349]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:11.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:11 smithi184 ceph-mon[100622]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:11 smithi184 ceph-mon[102844]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:40:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:11.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:11.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:11.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:13 smithi196 ceph-mon[76349]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:13 smithi184 ceph-mon[100622]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:13 smithi184 ceph-mon[102844]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:15 smithi196 ceph-mon[76349]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:15 smithi184 ceph-mon[100622]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:15 smithi184 ceph-mon[102844]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:16.484 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:16 smithi184 ceph-mon[100622]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:16.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:16 smithi184 ceph-mon[102844]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:16 smithi196 ceph-mon[76349]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:18 smithi184 ceph-mon[100622]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:18 smithi184 ceph-mon[102844]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:18 smithi196 ceph-mon[76349]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[100622]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[102844]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:20 smithi196 ceph-mon[76349]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:21.723 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:40:22.002 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:21.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:21.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:22.003 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:21.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:22.252 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:40:22.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 6h - - 2024-09-21T02:40:22.252 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 6h - - 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 3s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 1s ago 6h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 3s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 3s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 3s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 1s ago 6h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 3s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:40:22.253 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 3s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 3s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 6h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:40:22.254 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:40:22.255 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:40:22.562 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:40:22.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:40:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:40:22.564 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:40:22.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:40:22.832 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:40:22.833 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:40:22.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:40:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:22 smithi196 ceph-mon[76349]: from='client.81722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:22 smithi196 ceph-mon[76349]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:22 smithi196 ceph-mon[76349]: from='client.81728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3753732389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:23.141 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:23.141 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:23.141 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:23.141 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:23.141 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[100622]: from='client.81722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[100622]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[100622]: from='client.81728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3753732389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[102844]: from='client.81722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[102844]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[102844]: from='client.81728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:23.142 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3753732389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[100622]: from='client.71769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[100622]: from='client.81746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2725439809' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[102844]: from='client.71769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[102844]: from='client.81746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2725439809' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:23 smithi196 ceph-mon[76349]: from='client.71769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:23 smithi196 ceph-mon[76349]: from='client.81746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2725439809' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:24.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:24 smithi184 ceph-mon[100622]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:24.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:24 smithi184 ceph-mon[102844]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:24 smithi196 ceph-mon[76349]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:26 smithi184 ceph-mon[100622]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:26 smithi184 ceph-mon[102844]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:26 smithi196 ceph-mon[76349]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:28.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:28 smithi184 ceph-mon[100622]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:28 smithi184 ceph-mon[102844]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:29.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:28 smithi196 ceph-mon[76349]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:30 smithi184 ceph-mon[100622]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:30 smithi184 ceph-mon[102844]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:30 smithi196 ceph-mon[76349]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:32.213 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:40:32.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:31.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:32.214 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:31.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:32.214 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:31.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:32 smithi196 ceph-mon[76349]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:32 smithi184 ceph-mon[100622]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:32 smithi184 ceph-mon[102844]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:34 smithi184 ceph-mon[100622]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:34 smithi184 ceph-mon[102844]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:34 smithi196 ceph-mon[76349]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:36 smithi184 ceph-mon[100622]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:36 smithi184 ceph-mon[102844]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:36 smithi196 ceph-mon[76349]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[100622]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[102844]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:38 smithi196 ceph-mon[76349]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:39.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[102844]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[100622]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:40 smithi196 ceph-mon[76349]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:40:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:41.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:41.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:41.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:42 smithi184 ceph-mon[100622]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:42 smithi184 ceph-mon[102844]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:42 smithi196 ceph-mon[76349]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:45.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:45 smithi184 ceph-mon[102844]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:45 smithi184 ceph-mon[100622]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:45 smithi196 ceph-mon[76349]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:47.222 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:47 smithi184 ceph-mon[100622]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:47.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:47 smithi184 ceph-mon[102844]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:47 smithi196 ceph-mon[76349]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:49 smithi184 ceph-mon[100622]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:49 smithi184 ceph-mon[102844]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:49 smithi196 ceph-mon[76349]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:51 smithi184 ceph-mon[100622]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:51 smithi184 ceph-mon[102844]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:51 smithi196 ceph-mon[76349]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:40:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:40:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:40:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:51.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:51.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:40:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:40:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:40:51.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:40:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:53 smithi184 ceph-mon[100622]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:53 smithi184 ceph-mon[102844]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:53 smithi196 ceph-mon[76349]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:53.440 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:40:53.967 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:40:53.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 6h - - 2024-09-21T02:40:53.967 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 6h - - 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 13s ago 6h 96.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 13s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 13s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:40:53.968 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 13s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 13s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 13s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 13s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 6h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:40:53.969 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 13s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:40:54.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:40:54.280 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:40:54.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:40:54.548 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:40:54.549 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:40:54.855 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:54.856 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:54.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:54.856 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:54.856 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:55.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: from='client.71793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: from='client.81770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2819801911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4281676368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: from='client.71793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: from='client.81770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2819801911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4281676368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: from='client.71793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: from='client.81770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2819801911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4281676368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:56 smithi196 ceph-mon[76349]: from='client.71817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:56.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:56 smithi184 ceph-mon[100622]: from='client.71817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:56.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:56 smithi184 ceph-mon[102844]: from='client.71817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:57 smithi196 ceph-mon[76349]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:57 smithi184 ceph-mon[100622]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:57 smithi184 ceph-mon[102844]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[102844]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[100622]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:40:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:59 smithi196 ceph-mon[76349]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:40:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:40:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:00.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:01 smithi196 ceph-mon[76349]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[100622]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[102844]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:01.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:41:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:01.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:01.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:01.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:03 smithi196 ceph-mon[76349]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:03 smithi184 ceph-mon[100622]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:03 smithi184 ceph-mon[102844]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:05 smithi184 ceph-mon[100622]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:05 smithi184 ceph-mon[102844]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:05 smithi196 ceph-mon[76349]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:07.209 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:07 smithi184 ceph-mon[100622]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:07.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:07 smithi184 ceph-mon[102844]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:07 smithi196 ceph-mon[76349]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:09 smithi196 ceph-mon[76349]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:09 smithi184 ceph-mon[100622]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:09 smithi184 ceph-mon[102844]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:11 smithi196 ceph-mon[76349]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:11 smithi184 ceph-mon[100622]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:11 smithi184 ceph-mon[102844]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:41:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:11.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:11.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:11.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:13 smithi196 ceph-mon[76349]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:13 smithi184 ceph-mon[100622]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:13 smithi184 ceph-mon[102844]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:15 smithi196 ceph-mon[76349]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:15 smithi184 ceph-mon[100622]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:15 smithi184 ceph-mon[102844]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:17.377 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:17 smithi196 ceph-mon[76349]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:17 smithi184 ceph-mon[100622]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:17 smithi184 ceph-mon[102844]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:19 smithi196 ceph-mon[76349]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[100622]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[102844]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:21 smithi196 ceph-mon[76349]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[100622]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[102844]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:41:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:21.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:21.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:21.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:23 smithi196 ceph-mon[76349]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:23.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:23 smithi184 ceph-mon[100622]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:23 smithi184 ceph-mon[102844]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:24 smithi184 ceph-mon[100622]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:24 smithi184 ceph-mon[102844]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:24 smithi196 ceph-mon[76349]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:25.153 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:41:25.686 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 6h - - 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 6h - - 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 6s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 4s ago 6h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 6s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 6s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 6s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 6s ago 6h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 6s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 4s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 6s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:41:25.687 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 6s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 6s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 6h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:41:25.688 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:41:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:41:26.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:41:26.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:41:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:41:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:41:26.291 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:41:26.606 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:41:26.606 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:26.606 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:26.606 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:26.606 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:27.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: from='client.81794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.136 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2474410299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2657468630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: from='client.81794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2474410299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:27.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2657468630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: from='client.81794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2474410299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2657468630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:28.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:27 smithi184 ceph-mon[100622]: from='client.81818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:28.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:28.100 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:27 smithi184 ceph-mon[102844]: from='client.81818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:28.100 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:27 smithi196 ceph-mon[76349]: from='client.81818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:28 smithi196 ceph-mon[76349]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:28 smithi184 ceph-mon[100622]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:28 smithi184 ceph-mon[102844]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:30 smithi196 ceph-mon[76349]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:30 smithi184 ceph-mon[100622]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:30 smithi184 ceph-mon[102844]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:41:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:31.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:31.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:31.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:33.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:32 smithi196 ceph-mon[76349]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:32 smithi184 ceph-mon[100622]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:32 smithi184 ceph-mon[102844]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:34 smithi196 ceph-mon[76349]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:34 smithi184 ceph-mon[100622]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:34 smithi184 ceph-mon[102844]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:36 smithi196 ceph-mon[76349]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:36 smithi184 ceph-mon[100622]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:36 smithi184 ceph-mon[102844]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:38 smithi196 ceph-mon[76349]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:39.225 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:38 smithi184 ceph-mon[100622]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:39.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:38 smithi184 ceph-mon[102844]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[100622]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:40 smithi184 ceph-mon[102844]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:40 smithi196 ceph-mon[76349]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:41:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:41:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:41.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:41.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:41.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:42 smithi184 ceph-mon[100622]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:42 smithi184 ceph-mon[102844]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:42 smithi196 ceph-mon[76349]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:44 smithi184 ceph-mon[100622]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:44 smithi184 ceph-mon[102844]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:44 smithi196 ceph-mon[76349]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:47.277 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:46 smithi184 ceph-mon[100622]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:47.277 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:46 smithi184 ceph-mon[102844]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:46 smithi196 ceph-mon[76349]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:48.149 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:48.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:48 smithi184 ceph-mon[100622]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:48 smithi184 ceph-mon[102844]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:48 smithi196 ceph-mon[76349]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:50 smithi184 ceph-mon[100622]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:50 smithi184 ceph-mon[102844]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:50 smithi196 ceph-mon[76349]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:52.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:41:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:41:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:41:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:51.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:51.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:41:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:41:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:41:51.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:41:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:52 smithi184 ceph-mon[100622]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:41:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:52 smithi184 ceph-mon[102844]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:41:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:52 smithi196 ceph-mon[76349]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:41:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:54 smithi184 ceph-mon[100622]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:54 smithi184 ceph-mon[102844]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:54 smithi196 ceph-mon[76349]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:56.902 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:41:57.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:56 smithi196 ceph-mon[76349]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:57.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:56 smithi184 ceph-mon[100622]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:56 smithi184 ceph-mon[102844]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 6h - - 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 6h - - 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 18s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 16s ago 6h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 18s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 16s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:41:57.428 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 18s ago 6h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 16s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 6h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 18s ago 6h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 16s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 16s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 16s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 16s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 16s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 16s ago 6h 74.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:41:57.429 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:41:57.430 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:41:57.430 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:41:57.430 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 16s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:41:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:41:57.744 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:41:58.014 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:41:58.014 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:41:58.014 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:41:58.014 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:41:58.014 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:41:58.015 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:41:58.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:58.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[100622]: from='client.81830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:58.015 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2145474077' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:58.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:58.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[102844]: from='client.81830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:58.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2145474077' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:58.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:58.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:57 smithi196 ceph-mon[76349]: from='client.81830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:58.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2145474077' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:58.323 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:41:58.323 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:58.323 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:58.323 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:58.323 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[100622]: from='client.81836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[100622]: from='client.71877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[100622]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[100622]: from='client.81854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2227361717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[102844]: from='client.81836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[102844]: from='client.71877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[102844]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[102844]: from='client.81854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2227361717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:58 smithi196 ceph-mon[76349]: from='client.81836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:58 smithi196 ceph-mon[76349]: from='client.71877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:58 smithi196 ceph-mon[76349]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:58 smithi196 ceph-mon[76349]: from='client.81854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2227361717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:41:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:41:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:01.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:00 smithi184 ceph-mon[100622]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:00 smithi184 ceph-mon[102844]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:00 smithi196 ceph-mon[76349]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:01.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:01.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:01.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:03.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:03 smithi184 ceph-mon[100622]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:03 smithi184 ceph-mon[102844]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:03 smithi196 ceph-mon[76349]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:05 smithi184 ceph-mon[100622]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:05 smithi184 ceph-mon[102844]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:05 smithi196 ceph-mon[76349]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:07.281 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:07 smithi184 ceph-mon[100622]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:07.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:07 smithi184 ceph-mon[102844]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:07 smithi196 ceph-mon[76349]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:08.211 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:08.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:09.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:09 smithi184 ceph-mon[100622]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:09.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:09 smithi184 ceph-mon[102844]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:09 smithi196 ceph-mon[76349]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:11 smithi196 ceph-mon[76349]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:11 smithi184 ceph-mon[100622]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:11 smithi184 ceph-mon[102844]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:42:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:11.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:11.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:11.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:13 smithi196 ceph-mon[76349]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:13.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:13 smithi184 ceph-mon[100622]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:13 smithi184 ceph-mon[102844]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:15 smithi196 ceph-mon[76349]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:15 smithi184 ceph-mon[100622]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:15 smithi184 ceph-mon[102844]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:17 smithi196 ceph-mon[76349]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:17 smithi184 ceph-mon[100622]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:17.681 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:17 smithi184 ceph-mon[102844]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:18.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:19 smithi196 ceph-mon[76349]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:19.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:19 smithi184 ceph-mon[100622]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:19.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:19 smithi184 ceph-mon[102844]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[100622]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:20 smithi184 ceph-mon[102844]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:20 smithi196 ceph-mon[76349]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[100622]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:42:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:22 smithi184 ceph-mon[102844]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:21.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:21.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:21.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:22 smithi196 ceph-mon[76349]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:24 smithi196 ceph-mon[76349]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:24 smithi184 ceph-mon[100622]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:24 smithi184 ceph-mon[102844]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:26 smithi196 ceph-mon[76349]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:26 smithi184 ceph-mon[102844]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:26 smithi184 ceph-mon[100622]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:28.624 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:42:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:28 smithi196 ceph-mon[76349]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:29.193 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:28 smithi184 ceph-mon[100622]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:29.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:28 smithi184 ceph-mon[102844]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:29.216 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:42:29.216 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 6h - - 2024-09-21T02:42:29.216 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 6h - - 2024-09-21T02:42:29.216 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 9s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 92.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 9s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 9s ago 7h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 9s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 6h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 9s ago 6h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 9s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:42:29.217 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 9s ago 6h 84.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 9s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 8s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 8s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 8s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 76.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:42:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 8s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:42:29.540 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:42:29.541 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:42:29.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:42:29.835 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:42:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:29 smithi196 ceph-mon[76349]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:29 smithi196 ceph-mon[76349]: from='client.81872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4196404925' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:30.156 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:42:30.156 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:30.156 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:30.156 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:30.156 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:30.156 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[100622]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[100622]: from='client.81872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4196404925' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[102844]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[102844]: from='client.81872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:30.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4196404925' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[100622]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[100622]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[100622]: from='client.81890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/726206378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[102844]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[102844]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[102844]: from='client.81890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/726206378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:30 smithi196 ceph-mon[76349]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:30 smithi196 ceph-mon[76349]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:30 smithi196 ceph-mon[76349]: from='client.81890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/726206378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:32.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:31.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:32.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:31.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:32.222 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:31.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:32.222 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:42:33.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:32 smithi184 ceph-mon[100622]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:32 smithi184 ceph-mon[102844]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:32 smithi196 ceph-mon[76349]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:34 smithi184 ceph-mon[100622]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:34 smithi184 ceph-mon[102844]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:34 smithi196 ceph-mon[76349]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:37.075 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:36 smithi196 ceph-mon[76349]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:36 smithi184 ceph-mon[100622]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:36 smithi184 ceph-mon[102844]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:38.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:39.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:38 smithi196 ceph-mon[76349]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:38 smithi184 ceph-mon[100622]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:38 smithi184 ceph-mon[102844]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:40 smithi196 ceph-mon[76349]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[102844]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:40 smithi184 ceph-mon[100622]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:42.274 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:42:42.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:41.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:42.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:41.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:42.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:41.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[102844]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:42 smithi184 ceph-mon[100622]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:42:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:42 smithi196 ceph-mon[76349]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:44 smithi196 ceph-mon[76349]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:44 smithi184 ceph-mon[100622]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:44 smithi184 ceph-mon[102844]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:47.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:46 smithi196 ceph-mon[76349]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:46 smithi184 ceph-mon[100622]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:46 smithi184 ceph-mon[102844]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:48.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:48.505 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:48.505 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:48 smithi196 ceph-mon[76349]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:49.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:48 smithi184 ceph-mon[100622]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:49.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:48 smithi184 ceph-mon[102844]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:51.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:50 smithi184 ceph-mon[102844]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:50 smithi184 ceph-mon[100622]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:50 smithi196 ceph-mon[76349]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:42:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:42:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:42:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:51.971Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:51.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:42:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:42:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:42:51.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:42:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:52 smithi184 ceph-mon[102844]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:52 smithi184 ceph-mon[100622]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:52 smithi196 ceph-mon[76349]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:54 smithi184 ceph-mon[102844]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:54 smithi184 ceph-mon[100622]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:54 smithi196 ceph-mon[76349]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:56 smithi184 ceph-mon[100622]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:56 smithi184 ceph-mon[102844]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:57.334 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:56 smithi196 ceph-mon[76349]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:58.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:58 smithi184 ceph-mon[100622]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:58 smithi184 ceph-mon[102844]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:58 smithi196 ceph-mon[76349]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:42:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:42:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:00.448 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:43:00.979 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 6h - - 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 6h - - 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 19s ago 6h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 19s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 7h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 7h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 19s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 19s ago 6h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:43:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 6h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 19s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 19s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 19s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 19s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 19s ago 6h 76.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:43:00.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 19s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:43:01.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:00 smithi184 ceph-mon[102844]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:43:01.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:43:01.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:43:01.294 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:00 smithi184 ceph-mon[100622]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:00 smithi196 ceph-mon[76349]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:43:01.569 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:43:01.570 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:43:01.570 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:43:01.885 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:43:01.885 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:01.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:01.885 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:01.885 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:43:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='client.71937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='client.81908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1414491182' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2685156104' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:01.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:01.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:01.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='client.71937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='client.81908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1414491182' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2685156104' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='client.71937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='client.81908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1414491182' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2685156104' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:03.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:02 smithi184 ceph-mon[100622]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:02 smithi184 ceph-mon[100622]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:02 smithi184 ceph-mon[102844]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:02 smithi184 ceph-mon[102844]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:02 smithi196 ceph-mon[76349]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:02 smithi196 ceph-mon[76349]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:04 smithi184 ceph-mon[100622]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:04 smithi184 ceph-mon[102844]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:04 smithi196 ceph-mon[76349]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:06 smithi184 ceph-mon[100622]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:06 smithi184 ceph-mon[102844]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:06 smithi196 ceph-mon[76349]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:09.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:08 smithi184 ceph-mon[100622]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:09.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:08 smithi184 ceph-mon[102844]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:08 smithi196 ceph-mon[76349]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:11.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:10 smithi184 ceph-mon[100622]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:11.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:10 smithi184 ceph-mon[102844]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:10 smithi196 ceph-mon[76349]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:43:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:11.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:11.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:11.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:12 smithi184 ceph-mon[100622]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:12 smithi184 ceph-mon[102844]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:12 smithi196 ceph-mon[76349]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:14 smithi184 ceph-mon[100622]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:14 smithi184 ceph-mon[102844]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:14 smithi196 ceph-mon[76349]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:16 smithi184 ceph-mon[102844]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:16 smithi184 ceph-mon[100622]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:16 smithi196 ceph-mon[76349]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:18.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:18.297 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:19.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:18 smithi196 ceph-mon[76349]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:18 smithi184 ceph-mon[102844]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:18 smithi184 ceph-mon[100622]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:20 smithi196 ceph-mon[76349]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[102844]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[100622]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:21.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:43:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:21.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:21.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:21.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[102844]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:22 smithi184 ceph-mon[100622]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:22 smithi196 ceph-mon[76349]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:24 smithi184 ceph-mon[100622]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:24 smithi184 ceph-mon[102844]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:24 smithi196 ceph-mon[76349]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:26 smithi184 ceph-mon[100622]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:26 smithi184 ceph-mon[102844]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:26 smithi196 ceph-mon[76349]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:28.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:28 smithi184 ceph-mon[100622]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:28 smithi184 ceph-mon[102844]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:28 smithi196 ceph-mon[76349]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:29.987 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:29.987 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:31.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:30 smithi184 ceph-mon[100622]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:31.039 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:30 smithi184 ceph-mon[102844]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:30 smithi196 ceph-mon[76349]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:32.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:31.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:32.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:31.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:32.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:31.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:32.183 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:43:32.184 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:43:32.728 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:43:32.728 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 7h - - 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 12s ago 6h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 12s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 12s ago 7h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 12s ago 7h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 12s ago 6h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 12s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:43:32.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 12s ago 6h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 12s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:43:32.730 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:43:33.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:32 smithi184 ceph-mon[100622]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:33.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:32 smithi184 ceph-mon[102844]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:33.050 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:43:33.050 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:43:33.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:43:33.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:33.050 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:43:33.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:43:33.321 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:43:33.321 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:43:33.321 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:43:33.321 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:43:33.321 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:43:33.322 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:43:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:32 smithi196 ceph-mon[76349]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:43:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:33.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:33.636 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:33.636 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[102844]: from='client.81938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[102844]: from='client.81944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[102844]: from='client.81950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2549691932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3336344367' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[100622]: from='client.81938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[100622]: from='client.81944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[100622]: from='client.81950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2549691932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:34.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3336344367' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:33 smithi196 ceph-mon[76349]: from='client.81938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:33 smithi196 ceph-mon[76349]: from='client.81944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:33 smithi196 ceph-mon[76349]: from='client.81950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2549691932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3336344367' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:34 smithi184 ceph-mon[102844]: from='client.71997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:34 smithi184 ceph-mon[102844]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:34 smithi184 ceph-mon[100622]: from='client.71997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:34 smithi184 ceph-mon[100622]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:34 smithi196 ceph-mon[76349]: from='client.71997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:34 smithi196 ceph-mon[76349]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:36 smithi184 ceph-mon[100622]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:36 smithi184 ceph-mon[102844]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:36 smithi196 ceph-mon[76349]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:38.343 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:38 smithi184 ceph-mon[100622]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:38 smithi184 ceph-mon[102844]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:38 smithi196 ceph-mon[76349]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[100622]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[102844]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:40 smithi196 ceph-mon[76349]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:41.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:43:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:41.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:41.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:41.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[100622]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:42 smithi184 ceph-mon[102844]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:43:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:42 smithi196 ceph-mon[76349]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:44 smithi184 ceph-mon[100622]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:44 smithi184 ceph-mon[102844]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:44 smithi196 ceph-mon[76349]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:46 smithi184 ceph-mon[100622]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:46 smithi184 ceph-mon[102844]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:46 smithi196 ceph-mon[76349]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:48 smithi184 ceph-mon[100622]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:48 smithi184 ceph-mon[102844]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:48 smithi196 ceph-mon[76349]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:51.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:50 smithi184 ceph-mon[100622]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:50 smithi184 ceph-mon[102844]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:50 smithi196 ceph-mon[76349]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:52.286 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:43:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:43:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:43:52.286 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:51.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:52.286 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:51.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:43:52.286 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:43:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:43:51.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:43:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:52 smithi184 ceph-mon[100622]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:52 smithi184 ceph-mon[102844]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:52 smithi196 ceph-mon[76349]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:54 smithi184 ceph-mon[100622]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:54 smithi184 ceph-mon[102844]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:54 smithi196 ceph-mon[76349]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:57.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:56 smithi184 ceph-mon[100622]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:57.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:56 smithi184 ceph-mon[102844]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:56 smithi196 ceph-mon[76349]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:58.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:59.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:58 smithi196 ceph-mon[76349]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:59.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:58 smithi184 ceph-mon[100622]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:58 smithi184 ceph-mon[102844]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:43:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:43:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:43:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:01 smithi196 ceph-mon[76349]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[100622]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[102844]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:44:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:01.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:01.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:01.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[102844]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:02 smithi184 ceph-mon[100622]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:03.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:02 smithi196 ceph-mon[76349]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:03.933 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:44:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:44:04.461 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 7h - - 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 7h - - 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 4s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 2s ago 6h 93.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 4s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 2s ago 6h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 4s ago 7h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 4s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 2s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 4s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 4s ago 6h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 2s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 4s ago 6h 96.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:44:04.462 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 6h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 2s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 2s ago 6h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 2s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 2s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 2s ago 6h 78.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:44:04.463 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 2s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:44:04.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:04.773 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:44:04.773 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:44:04.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:44:04.773 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:44:05.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:44:05.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:44:05.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[102844]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:05.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[102844]: from='client.72009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:05.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2400700923' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:05.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[100622]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:05.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[100622]: from='client.72009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:05.044 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2400700923' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:05.354 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:44:05.355 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:05.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:05.355 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:05.355 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:04 smithi196 ceph-mon[76349]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:04 smithi196 ceph-mon[76349]: from='client.72009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2400700923' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[100622]: from='client.81980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[100622]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[100622]: from='client.72033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1225780841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[102844]: from='client.81980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[102844]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[102844]: from='client.72033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1225780841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:05 smithi196 ceph-mon[76349]: from='client.81980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:05 smithi196 ceph-mon[76349]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:05 smithi196 ceph-mon[76349]: from='client.72033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:06.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1225780841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:06 smithi184 ceph-mon[100622]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:06 smithi184 ceph-mon[102844]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:06 smithi196 ceph-mon[76349]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:09.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:08 smithi184 ceph-mon[100622]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:44:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:08 smithi184 ceph-mon[102844]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:44:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:08 smithi196 ceph-mon[76349]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:44:11.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:10 smithi184 ceph-mon[100622]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:44:11.260 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:10 smithi184 ceph-mon[102844]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:44:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:10 smithi196 ceph-mon[76349]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:44:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:44:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:11.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:11.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:11.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:12 smithi184 ceph-mon[100622]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:44:13.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:12 smithi184 ceph-mon[102844]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:44:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:12 smithi196 ceph-mon[76349]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:44:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:14 smithi184 ceph-mon[100622]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:44:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:14 smithi184 ceph-mon[102844]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:44:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:14 smithi196 ceph-mon[76349]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:44:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:17.047 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:16 smithi184 ceph-mon[100622]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:44:17.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:16 smithi184 ceph-mon[102844]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:44:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:16 smithi196 ceph-mon[76349]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:44:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:17 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:17 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:17 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:18 smithi184 ceph-mon[100622]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T02:44:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:18 smithi184 ceph-mon[102844]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T02:44:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:18 smithi196 ceph-mon[76349]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T02:44:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:20 smithi196 ceph-mon[76349]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T02:44:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[102844]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[100622]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 377 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:44:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:21.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:21.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:21.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[100622]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 381 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[102844]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 381 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:22 smithi196 ceph-mon[76349]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 381 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:24 smithi184 ceph-mon[100622]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:44:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:24 smithi184 ceph-mon[102844]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:44:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:24 smithi196 ceph-mon[76349]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:44:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:26 smithi184 ceph-mon[100622]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:26 smithi184 ceph-mon[102844]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:26 smithi196 ceph-mon[76349]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:44:28.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:28.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:28 smithi196 ceph-mon[76349]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:44:29.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:28 smithi184 ceph-mon[100622]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:44:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:28 smithi184 ceph-mon[102844]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:44:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:31.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:30 smithi184 ceph-mon[100622]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:44:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:30 smithi184 ceph-mon[102844]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:44:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:30 smithi196 ceph-mon[76349]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:44:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:44:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:31.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:31.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:32.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:31.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:33.137 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:32 smithi184 ceph-mon[100622]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:44:33.137 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:32 smithi184 ceph-mon[102844]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:44:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:32 smithi196 ceph-mon[76349]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:44:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:34 smithi184 ceph-mon[100622]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:44:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:34 smithi184 ceph-mon[102844]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:44:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:34 smithi196 ceph-mon[76349]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-09-21T02:44:35.649 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 7h - - 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 15s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 14s ago 6h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 15s ago 7h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:44:36.176 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 15s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 6h 84.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 6h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 6h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 6h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 6h 78.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:44:36.177 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:44:36.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:44:36.486 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:44:36.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:44:36.486 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:44:36.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:44:36.487 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:44:36.755 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:44:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:44:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:44:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:44:36.756 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[102844]: from='client.72045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[102844]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[102844]: from='client.82016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/576273662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[100622]: from='client.72045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[100622]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[100622]: from='client.82016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/576273662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:37.099 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:44:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:37.100 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:36 smithi196 ceph-mon[76349]: from='client.72045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:36 smithi196 ceph-mon[76349]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:36 smithi196 ceph-mon[76349]: from='client.82016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/576273662' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[100622]: from='client.82022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[100622]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1540731551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[102844]: from='client.82022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[102844]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1540731551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:37 smithi196 ceph-mon[76349]: from='client.82022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:37 smithi196 ceph-mon[76349]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1540731551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:38 smithi184 ceph-mon[100622]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:38 smithi184 ceph-mon[102844]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:38 smithi196 ceph-mon[76349]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:41.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:40 smithi196 ceph-mon[76349]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:41.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[100622]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[102844]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:44:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:44:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:41.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:41.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:41.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[100622]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[102844]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:43 smithi196 ceph-mon[76349]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:44:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:44:45.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:45 smithi184 ceph-mon[100622]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:44:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:45 smithi184 ceph-mon[102844]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:44:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:45 smithi196 ceph-mon[76349]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:44:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:47 smithi184 ceph-mon[102844]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:44:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:47 smithi184 ceph-mon[100622]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:44:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:47 smithi196 ceph-mon[76349]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:44:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:49 smithi184 ceph-mon[100622]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:44:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:49 smithi184 ceph-mon[102844]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:44:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:49 smithi196 ceph-mon[76349]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:44:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:51 smithi184 ceph-mon[100622]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:44:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:51 smithi184 ceph-mon[102844]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:44:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:51 smithi196 ceph-mon[76349]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:44:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:44:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:44:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:44:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:51.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:51.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:44:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:44:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:44:51.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:44:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:53 smithi184 ceph-mon[100622]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:44:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:53 smithi184 ceph-mon[102844]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:44:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:53 smithi196 ceph-mon[76349]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:44:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:55 smithi184 ceph-mon[100622]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:55 smithi184 ceph-mon[102844]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:55 smithi196 ceph-mon[76349]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:57 smithi184 ceph-mon[100622]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:57 smithi184 ceph-mon[102844]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:57 smithi196 ceph-mon[76349]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:58.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:44:59 smithi196 ceph-mon[76349]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:44:59 smithi184 ceph-mon[100622]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:44:59 smithi184 ceph-mon[102844]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:00.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:00.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:01.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:01 smithi196 ceph-mon[76349]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:01.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.537 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[100622]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[102844]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:02] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:45:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:01.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:01.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:01.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:03.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:03 smithi196 ceph-mon[76349]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:03.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[102844]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[100622]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:03.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:05.322 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:05 smithi184 ceph-mon[100622]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:05.323 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:05 smithi184 ceph-mon[102844]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:05 smithi196 ceph-mon[76349]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:07 smithi196 ceph-mon[76349]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:07.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:07 smithi184 ceph-mon[100622]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:07.398 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:07 smithi184 ceph-mon[102844]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:07.399 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 7h - - 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 7s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 5s ago 6h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 7s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:45:07.928 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 5s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 7s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 7s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 5s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 7s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 7s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 5s ago 6h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 7s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 7s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 6h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 6h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 5s ago 6h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 5s ago 6h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 5s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:45:07.929 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 5s ago 6h 78.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:45:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:45:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:45:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:45:07.930 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 5s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:45:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:45:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:45:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:45:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:45:08.245 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:45:08.245 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:08.245 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:45:08.517 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:45:08.518 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:45:08.827 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:45:08.827 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:08.827 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:08.828 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:08.828 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:09.077 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: from='client.82046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: from='client.72087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: from='client.82046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: from='client.72087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3664186029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:09.078 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2460893619' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: from='client.82046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: from='client.72087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3664186029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:09.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2460893619' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3664186029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2460893619' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:10 smithi196 ceph-mon[76349]: from='client.72105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:10 smithi184 ceph-mon[100622]: from='client.72105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:10 smithi184 ceph-mon[102844]: from='client.72105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:11 smithi196 ceph-mon[76349]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:11 smithi184 ceph-mon[100622]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:11 smithi184 ceph-mon[102844]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:12] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:45:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:11.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:11.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:11.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:13 smithi184 ceph-mon[100622]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:13 smithi184 ceph-mon[102844]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:13 smithi196 ceph-mon[76349]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:15 smithi196 ceph-mon[76349]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:15 smithi184 ceph-mon[100622]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:15 smithi184 ceph-mon[102844]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:17 smithi196 ceph-mon[76349]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:17 smithi184 ceph-mon[100622]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:17 smithi184 ceph-mon[102844]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:19 smithi196 ceph-mon[76349]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:19 smithi184 ceph-mon[100622]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:19 smithi184 ceph-mon[102844]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:21 smithi196 ceph-mon[76349]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[100622]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[102844]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:45:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:21.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:21.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:21.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:23 smithi196 ceph-mon[76349]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[100622]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[102844]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:25 smithi196 ceph-mon[76349]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:25 smithi184 ceph-mon[100622]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:25 smithi184 ceph-mon[102844]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:27 smithi196 ceph-mon[76349]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:27 smithi184 ceph-mon[100622]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:27 smithi184 ceph-mon[102844]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:29 smithi196 ceph-mon[76349]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:29 smithi184 ceph-mon[100622]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:29 smithi184 ceph-mon[102844]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:31 smithi184 ceph-mon[100622]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:31 smithi184 ceph-mon[102844]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:31 smithi196 ceph-mon[76349]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:45:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:31.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:31.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:31.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:33.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:33 smithi184 ceph-mon[100622]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:33.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:33 smithi184 ceph-mon[102844]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:33 smithi196 ceph-mon[76349]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:35 smithi184 ceph-mon[100622]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:35 smithi184 ceph-mon[102844]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:35 smithi196 ceph-mon[76349]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:37 smithi184 ceph-mon[100622]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:37 smithi184 ceph-mon[102844]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:37 smithi196 ceph-mon[76349]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:39.126 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:45:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:39 smithi184 ceph-mon[100622]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:39 smithi184 ceph-mon[100622]: from='client.72117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:39 smithi184 ceph-mon[102844]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:39 smithi184 ceph-mon[102844]: from='client.72117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:39 smithi196 ceph-mon[76349]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:39 smithi196 ceph-mon[76349]: from='client.72117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 7h - - 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 19s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 17s ago 6h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 17s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:45:39.655 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 19s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 17s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 19s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 19s ago 6h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 17s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 19s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 19s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 19s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 6h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 6h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:45:39.656 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 17s ago 6h 78.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:45:39.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:45:39.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:45:39.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:45:39.657 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:45:39.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:45:39.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:45:39.965 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:45:39.965 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:45:39.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:45:39.965 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:45:40.236 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:45:40.237 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:45:40.238 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:45:40.239 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:45:40.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[102844]: from='client.82088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[102844]: from='client.72129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.239 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[102844]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2600099854' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:40.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[100622]: from='client.82088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[100622]: from='client.72129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[100622]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2600099854' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:40.547 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:45:40.547 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:40.547 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:40.547 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:40.547 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:40 smithi196 ceph-mon[76349]: from='client.82088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:40 smithi196 ceph-mon[76349]: from='client.72129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:40 smithi196 ceph-mon[76349]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2600099854' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[100622]: from='client.72141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/823807133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[102844]: from='client.72141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/823807133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:41 smithi196 ceph-mon[76349]: from='client.72141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/823807133' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:42 smithi184 ceph-mon[100622]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:42 smithi184 ceph-mon[102844]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:45:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:41.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:41.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:41.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:42.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:42 smithi196 ceph-mon[76349]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:43.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:45:44.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:44 smithi196 ceph-mon[76349]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:44.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:44 smithi184 ceph-mon[100622]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:44 smithi184 ceph-mon[102844]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:47.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:46 smithi184 ceph-mon[100622]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:47.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:46 smithi184 ceph-mon[102844]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:46 smithi196 ceph-mon[76349]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:48 smithi184 ceph-mon[100622]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:48 smithi184 ceph-mon[102844]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:48 smithi196 ceph-mon[76349]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:51.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:50 smithi184 ceph-mon[100622]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:50 smithi184 ceph-mon[102844]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:50 smithi196 ceph-mon[76349]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:45:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:45:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:45:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:51.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:51.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:45:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:45:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:45:51.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:45:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:52 smithi184 ceph-mon[100622]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:52 smithi184 ceph-mon[102844]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:52 smithi196 ceph-mon[76349]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:55.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:54 smithi184 ceph-mon[100622]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:55.268 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:54 smithi184 ceph-mon[102844]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:54 smithi196 ceph-mon[76349]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:56 smithi184 ceph-mon[100622]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:57.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:56 smithi184 ceph-mon[102844]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:56 smithi196 ceph-mon[76349]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:58.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:58.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:58 smithi184 ceph-mon[100622]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:58 smithi184 ceph-mon[102844]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:59.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:58 smithi196 ceph-mon[76349]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:45:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:45:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:45:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[100622]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[102844]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:00 smithi196 ceph-mon[76349]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:02] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:46:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:01.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:01.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:01.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[100622]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[102844]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:02 smithi196 ceph-mon[76349]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:03.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:03.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:04 smithi184 ceph-mon[100622]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:04 smithi184 ceph-mon[102844]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:05.352 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:04 smithi196 ceph-mon[76349]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:06 smithi184 ceph-mon[100622]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:06 smithi184 ceph-mon[102844]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:06 smithi196 ceph-mon[76349]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:09 smithi184 ceph-mon[100622]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:09 smithi184 ceph-mon[102844]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:09 smithi196 ceph-mon[76349]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:10.841 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:46:11.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:11 smithi184 ceph-mon[100622]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:11 smithi184 ceph-mon[102844]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 7h - - 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 10s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 8s ago 6h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 10s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 8s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 10s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 8s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:46:11.372 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 10s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 10s ago 6h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 8s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 10s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 10s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 10s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 10s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 8s ago 6h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 8s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 8s ago 6h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:46:11.373 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 8s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:46:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:11 smithi196 ceph-mon[76349]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:11.681 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:46:11.681 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:46:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:46:11.683 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:46:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:46:11.950 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:46:12.260 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:46:12.260 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:12.260 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:12.261 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:12] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:46:12.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[100622]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[100622]: from='client.82124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/771542493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:12.261 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[102844]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[102844]: from='client.82124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/771542493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:12.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:11.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:12.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:11.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:12.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:11.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:12 smithi196 ceph-mon[76349]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:12 smithi196 ceph-mon[76349]: from='client.82124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/771542493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[100622]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[100622]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[100622]: from='client.72177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1805764370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[102844]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[102844]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[102844]: from='client.72177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1805764370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:13 smithi196 ceph-mon[76349]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:13 smithi196 ceph-mon[76349]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:13 smithi196 ceph-mon[76349]: from='client.72177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:13.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1805764370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:15.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:15 smithi184 ceph-mon[100622]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:15.231 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:15.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:15 smithi184 ceph-mon[102844]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:15.232 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:15 smithi196 ceph-mon[76349]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:17.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:17 smithi184 ceph-mon[100622]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:17 smithi184 ceph-mon[102844]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:17 smithi196 ceph-mon[76349]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:19.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:19 smithi184 ceph-mon[100622]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:19 smithi184 ceph-mon[102844]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:19 smithi196 ceph-mon[76349]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[100622]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[102844]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:21.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:21 smithi196 ceph-mon[76349]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:22] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:46:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:21.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:21.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:21.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[102844]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[100622]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:23 smithi196 ceph-mon[76349]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:25 smithi184 ceph-mon[100622]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:25 smithi184 ceph-mon[102844]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:25 smithi196 ceph-mon[76349]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:27 smithi184 ceph-mon[100622]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:27 smithi184 ceph-mon[102844]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:27 smithi196 ceph-mon[76349]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:28.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:29 smithi196 ceph-mon[76349]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:29 smithi184 ceph-mon[100622]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:29 smithi184 ceph-mon[102844]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:31 smithi196 ceph-mon[76349]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:31 smithi184 ceph-mon[100622]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:31 smithi184 ceph-mon[102844]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:46:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:31.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:31.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:31.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:33 smithi196 ceph-mon[76349]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:33.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:33 smithi184 ceph-mon[100622]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:33.510 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:33 smithi184 ceph-mon[102844]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:35 smithi196 ceph-mon[76349]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:35 smithi184 ceph-mon[100622]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:35 smithi184 ceph-mon[102844]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:37.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:37 smithi184 ceph-mon[102844]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:37.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:37 smithi184 ceph-mon[100622]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:37 smithi196 ceph-mon[76349]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:38.166 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:38.167 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:39 smithi196 ceph-mon[76349]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:39 smithi184 ceph-mon[100622]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:39 smithi184 ceph-mon[102844]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:41 smithi196 ceph-mon[76349]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:41 smithi184 ceph-mon[100622]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:41 smithi184 ceph-mon[102844]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:41.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:41.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:41.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:42.559 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 7h - - 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 7h - - 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 1s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 0s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:46:43.095 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 1s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 0s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 1s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 1s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 1s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 1s ago 6h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 0s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 1s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 1s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 1s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 1s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 0s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:46:43.096 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 0s ago 6h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:46:43.097 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 0s ago 6h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:46:43.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:46:43.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:46:43.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:46:43.097 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[100622]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[102844]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.408 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:46:43.408 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:46:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:46:43.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:46:43.410 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:46:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:43 smithi196 ceph-mon[76349]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:46:43.682 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:46:43.682 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:46:43.682 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:46:43.683 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:46:43.991 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:46:43.991 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:43.991 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:43.991 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:43.991 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[100622]: from='client.72189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[100622]: from='client.72195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[100622]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4292012401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1983930887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[102844]: from='client.72189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[102844]: from='client.72195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[102844]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4292012401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1983930887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:44.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:44 smithi196 ceph-mon[76349]: from='client.72189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:44 smithi196 ceph-mon[76349]: from='client.72195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:44 smithi196 ceph-mon[76349]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:44.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4292012401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:44.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1983930887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:45.337 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:45 smithi196 ceph-mon[76349]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:45.337 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:45 smithi196 ceph-mon[76349]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:45.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[100622]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[100622]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[102844]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[102844]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:47 smithi184 ceph-mon[100622]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:47 smithi184 ceph-mon[102844]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:47 smithi196 ceph-mon[76349]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:49 smithi184 ceph-mon[100622]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:49 smithi184 ceph-mon[102844]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:49 smithi196 ceph-mon[76349]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:51 smithi184 ceph-mon[100622]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:51 smithi184 ceph-mon[102844]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:51 smithi196 ceph-mon[76349]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:46:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:46:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:46:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:51.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:51.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:46:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:46:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:46:51.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:46:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:53 smithi184 ceph-mon[100622]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:53 smithi184 ceph-mon[102844]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:53 smithi196 ceph-mon[76349]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:54 smithi184 ceph-mon[100622]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:54 smithi184 ceph-mon[102844]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:54 smithi196 ceph-mon[76349]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:57.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:56 smithi196 ceph-mon[76349]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:57.282 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:56 smithi184 ceph-mon[100622]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:57.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:56 smithi184 ceph-mon[102844]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:57 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:57 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:57 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:58 smithi196 ceph-mon[76349]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:58 smithi184 ceph-mon[100622]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:58 smithi184 ceph-mon[102844]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:46:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:46:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:46:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:01.263 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:00 smithi184 ceph-mon[100622]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:01.264 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:00 smithi184 ceph-mon[102844]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:00 smithi196 ceph-mon[76349]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:02.253 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:47:02.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:01.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:02.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:01.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:02.253 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:01.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[102844]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:02 smithi184 ceph-mon[100622]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:02 smithi196 ceph-mon[76349]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:04 smithi196 ceph-mon[76349]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:04 smithi184 ceph-mon[100622]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:04 smithi184 ceph-mon[102844]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:06 smithi184 ceph-mon[100622]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:06 smithi184 ceph-mon[102844]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:06 smithi196 ceph-mon[76349]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:08 smithi184 ceph-mon[100622]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:08 smithi184 ceph-mon[102844]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:08 smithi196 ceph-mon[76349]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:11 smithi184 ceph-mon[100622]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:11 smithi184 ceph-mon[102844]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:11 smithi196 ceph-mon[76349]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:47:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:11.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:11.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:11.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:13 smithi184 ceph-mon[100622]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:13 smithi184 ceph-mon[102844]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:13 smithi196 ceph-mon[76349]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:14.289 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:47:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:47:14.815 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 7h - - 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 13s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 11s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 13s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 11s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 13s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 13s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 11s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 13s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 13s ago 6h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 11s ago 6h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 13s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 13s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 13s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:47:14.816 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 13s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 11s ago 6h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:47:14.817 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:47:15.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:47:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:47:15.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:47:15.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:47:15.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:15 smithi184 ceph-mon[100622]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:15.129 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:15.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:15 smithi184 ceph-mon[102844]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:15.129 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:15 smithi196 ceph-mon[76349]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:47:15.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:47:15.711 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:47:15.711 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:15.711 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:15.711 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:15.711 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[102844]: from='client.72225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[102844]: from='client.72231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[102844]: from='client.72237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2041679216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2303937238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[100622]: from='client.72225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[100622]: from='client.72231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[100622]: from='client.72237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2041679216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2303937238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:16 smithi196 ceph-mon[76349]: from='client.72225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:16 smithi196 ceph-mon[76349]: from='client.72231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:16 smithi196 ceph-mon[76349]: from='client.72237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2041679216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2303937238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:17.132 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:17 smithi184 ceph-mon[100622]: from='client.72249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:17.133 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:17 smithi184 ceph-mon[100622]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:17.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:17 smithi184 ceph-mon[102844]: from='client.72249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:17.133 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:17 smithi184 ceph-mon[102844]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:17 smithi196 ceph-mon[76349]: from='client.72249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:17 smithi196 ceph-mon[76349]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:18.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:18.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:19 smithi184 ceph-mon[100622]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:19 smithi184 ceph-mon[102844]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:19 smithi196 ceph-mon[76349]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:21 smithi196 ceph-mon[76349]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:21.392 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:21 smithi184 ceph-mon[100622]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:21.393 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:21 smithi184 ceph-mon[102844]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:47:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:21.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:21.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:21.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:22 smithi196 ceph-mon[76349]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[100622]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:22 smithi184 ceph-mon[102844]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:25 smithi196 ceph-mon[76349]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:25 smithi184 ceph-mon[102844]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:25 smithi184 ceph-mon[100622]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:27 smithi184 ceph-mon[100622]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:27 smithi184 ceph-mon[102844]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:27.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:27 smithi196 ceph-mon[76349]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:28.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:29 smithi196 ceph-mon[76349]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:29.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:29 smithi184 ceph-mon[100622]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:29 smithi184 ceph-mon[102844]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:31 smithi196 ceph-mon[76349]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:31 smithi184 ceph-mon[102844]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:31 smithi184 ceph-mon[100622]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:32] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:47:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:31.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:31.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:31.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:33 smithi196 ceph-mon[76349]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:33.487 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:33 smithi184 ceph-mon[100622]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:33.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:33 smithi184 ceph-mon[102844]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:35 smithi196 ceph-mon[76349]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:35 smithi184 ceph-mon[100622]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:35 smithi184 ceph-mon[102844]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:37 smithi184 ceph-mon[100622]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:37 smithi184 ceph-mon[102844]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:37 smithi196 ceph-mon[76349]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:39.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:39 smithi184 ceph-mon[100622]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:39.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:39 smithi184 ceph-mon[102844]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:39 smithi196 ceph-mon[76349]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:41 smithi196 ceph-mon[76349]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:41.520 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:41 smithi184 ceph-mon[100622]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:41.520 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:41 smithi184 ceph-mon[102844]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:42] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T02:47:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:41.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:41.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:41.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[100622]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:42 smithi184 ceph-mon[102844]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:42.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:42 smithi196 ceph-mon[76349]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[102844]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[100622]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:47:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:44 smithi196 ceph-mon[76349]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:46.006 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 7h - - 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 7h - - 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 5s ago 6h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (6h) 3s ago 6h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:47:46.532 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 3s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 3s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 5s ago 6h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 3s ago 6h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 3s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:47:46.533 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 3s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 3s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 3s ago 7h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:47:46.534 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 3s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:47:46.843 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:47:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:47:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:47:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:46.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:47:46.844 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:47:47.113 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:47:47.114 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:47:47.115 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:47:47.115 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[102844]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[102844]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/11773419' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[100622]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[100622]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:47.115 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/11773419' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:47.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:46 smithi196 ceph-mon[76349]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:47.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:46 smithi196 ceph-mon[76349]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:47.338 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/11773419' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:47.420 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:47:47.420 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:47.420 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:47.420 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:47.421 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:48.187 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:47 smithi196 ceph-mon[76349]: from='client.72267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.187 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:47 smithi196 ceph-mon[76349]: from='client.82238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.187 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:48.187 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:47 smithi196 ceph-mon[76349]: from='client.72285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.188 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2654633899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[102844]: from='client.72267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[102844]: from='client.82238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[102844]: from='client.72285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2654633899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[100622]: from='client.72267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[100622]: from='client.82238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[100622]: from='client.72285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:48.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2654633899' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:48 smithi196 ceph-mon[76349]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:48 smithi184 ceph-mon[102844]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:48 smithi184 ceph-mon[100622]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:51 smithi184 ceph-mon[102844]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:51 smithi184 ceph-mon[100622]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:51 smithi196 ceph-mon[76349]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:47:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:47:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:47:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:51.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:51.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:47:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:47:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:47:51.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:47:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:53 smithi184 ceph-mon[100622]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:53 smithi184 ceph-mon[102844]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:53 smithi196 ceph-mon[76349]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:55 smithi184 ceph-mon[100622]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:55 smithi184 ceph-mon[102844]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:55 smithi196 ceph-mon[76349]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:57 smithi184 ceph-mon[100622]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:57 smithi184 ceph-mon[102844]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:57 smithi196 ceph-mon[76349]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:58.278 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:58.279 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:58.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:59.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:47:59 smithi184 ceph-mon[100622]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:59.284 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:47:59 smithi184 ceph-mon[102844]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:47:59 smithi196 ceph-mon[76349]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:01 smithi184 ceph-mon[100622]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:01 smithi184 ceph-mon[102844]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:01 smithi196 ceph-mon[76349]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:48:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:01.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:01.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:01.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:02 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:02 smithi196 ceph-mon[76349]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[102844]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:03.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:02 smithi184 ceph-mon[100622]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:04 smithi196 ceph-mon[76349]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[100622]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:04 smithi184 ceph-mon[102844]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:07.232 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:06 smithi196 ceph-mon[76349]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:06 smithi184 ceph-mon[100622]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:06 smithi184 ceph-mon[102844]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:08.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:09.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:09 smithi196 ceph-mon[76349]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:09 smithi184 ceph-mon[100622]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:09 smithi184 ceph-mon[102844]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:11 smithi184 ceph-mon[100622]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:11 smithi184 ceph-mon[102844]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:11 smithi196 ceph-mon[76349]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:48:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:11.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:11.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:11.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:13 smithi184 ceph-mon[100622]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:13 smithi184 ceph-mon[102844]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:13 smithi196 ceph-mon[76349]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:48:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:48:13.81312539Z level=info msg="Completed cleanup jobs" duration=13.590506ms 2024-09-21T02:48:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:15 smithi184 ceph-mon[100622]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:15 smithi184 ceph-mon[102844]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:15 smithi196 ceph-mon[76349]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:17 smithi184 ceph-mon[100622]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:17 smithi184 ceph-mon[102844]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:17 smithi196 ceph-mon[76349]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:17.716 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 16s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 16s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 14s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 16s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 16s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:48:18.242 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 14s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 16s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (6h) 16s ago 6h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (6h) 14s ago 6h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 16s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 16s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 16s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 16s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 14s ago 7h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:48:18.243 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:48:18.244 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:48:18.244 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:48:18.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:48:18.558 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:48:18.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:48:18.833 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:48:18.834 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:48:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:48:19.157 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:19.158 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:19.158 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:19.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[100622]: from='client.72297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[100622]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:19.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[100622]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.158 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1709361177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:19.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[102844]: from='client.72297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[102844]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:19.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[102844]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1709361177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:19 smithi196 ceph-mon[76349]: from='client.72297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:19 smithi196 ceph-mon[76349]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:19 smithi196 ceph-mon[76349]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1709361177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[100622]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[100622]: from='client.82286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/577788208' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[102844]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[102844]: from='client.82286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/577788208' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:20 smithi196 ceph-mon[76349]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:20 smithi196 ceph-mon[76349]: from='client.82286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/577788208' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:21.282 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:21 smithi184 ceph-mon[100622]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:21.282 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:21 smithi184 ceph-mon[102844]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:21 smithi196 ceph-mon[76349]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:48:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:21.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:21.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:21.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[102844]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:22 smithi184 ceph-mon[100622]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:23.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:23.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:22 smithi196 ceph-mon[76349]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[102844]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:24 smithi184 ceph-mon[100622]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:24 smithi196 ceph-mon[76349]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:27 smithi184 ceph-mon[100622]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:27 smithi184 ceph-mon[102844]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:27 smithi196 ceph-mon[76349]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:29 smithi196 ceph-mon[76349]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:29 smithi184 ceph-mon[100622]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:29 smithi184 ceph-mon[102844]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:31 smithi184 ceph-mon[100622]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:31 smithi184 ceph-mon[102844]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:31 smithi196 ceph-mon[76349]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:48:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:31.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:31.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:31 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:31.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:33.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:33 smithi184 ceph-mon[100622]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:33.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:33 smithi184 ceph-mon[102844]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:33 smithi196 ceph-mon[76349]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:35 smithi184 ceph-mon[100622]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:35 smithi184 ceph-mon[102844]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:35 smithi196 ceph-mon[76349]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:37 smithi184 ceph-mon[102844]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:37 smithi184 ceph-mon[100622]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:37 smithi196 ceph-mon[76349]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:39 smithi184 ceph-mon[100622]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:39 smithi184 ceph-mon[102844]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:39 smithi196 ceph-mon[76349]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:41 smithi184 ceph-mon[100622]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:41.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:41 smithi184 ceph-mon[102844]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:41 smithi196 ceph-mon[76349]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:48:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:41.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:41.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:41 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:41.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:42 smithi196 ceph-mon[76349]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[100622]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:42 smithi184 ceph-mon[102844]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:44 smithi196 ceph-mon[76349]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[100622]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[102844]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:47 smithi184 ceph-mon[100622]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:47 smithi184 ceph-mon[102844]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:47 smithi196 ceph-mon[76349]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:48.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:49 smithi184 ceph-mon[100622]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:49 smithi184 ceph-mon[102844]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:49.337 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:49 smithi196 ceph-mon[76349]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:49.454 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 7h - - 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 7h - - 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (6h) 8s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 6s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 6s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 7h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 6s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:48:49.980 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 8s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 6s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 6s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 6s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 6s ago 7h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:48:49.981 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 6s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:48:50.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:48:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:48:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:48:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:48:50.291 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:48:50.563 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:48:50.564 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:48:50.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:48:50.873 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:48:50.873 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:50.873 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:50.873 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:50.873 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: from='client.82298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: from='client.82304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3765264926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1623232752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: from='client.82298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: from='client.82304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3765264926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1623232752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: from='client.82298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: from='client.82304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3765264926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1623232752' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:52 smithi184 ceph-mon[100622]: from='client.82322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:48:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:48:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:48:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:52 smithi184 ceph-mon[102844]: from='client.82322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:51.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:51.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:48:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:48:51 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:48:51.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:48:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:52 smithi196 ceph-mon[76349]: from='client.82322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:53 smithi184 ceph-mon[100622]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:53 smithi184 ceph-mon[102844]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:53 smithi196 ceph-mon[76349]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:55 smithi184 ceph-mon[100622]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:55.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:55 smithi184 ceph-mon[102844]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:55 smithi196 ceph-mon[76349]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:57 smithi184 ceph-mon[100622]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:57 smithi184 ceph-mon[102844]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:57 smithi196 ceph-mon[76349]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:48:59 smithi184 ceph-mon[100622]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:48:59 smithi184 ceph-mon[102844]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:48:59 smithi196 ceph-mon[76349]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:00.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:00.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:01.321 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:01 smithi184 ceph-mon[100622]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:01.321 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:01 smithi184 ceph-mon[102844]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:01 smithi196 ceph-mon[76349]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:49:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:01.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:01.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:01 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:01.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[102844]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[100622]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:03 smithi196 ceph-mon[76349]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[100622]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:04.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:04 smithi184 ceph-mon[102844]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:05.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:05.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:04 smithi196 ceph-mon[76349]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:07 smithi184 ceph-mon[100622]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:07 smithi184 ceph-mon[102844]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:07 smithi196 ceph-mon[76349]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:09 smithi196 ceph-mon[76349]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:09 smithi184 ceph-mon[100622]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:09 smithi184 ceph-mon[102844]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:11.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:11 smithi196 ceph-mon[76349]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:11 smithi184 ceph-mon[100622]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:11 smithi184 ceph-mon[102844]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:49:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:11.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:11.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:11 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:11.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:13 smithi196 ceph-mon[76349]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:13 smithi184 ceph-mon[100622]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:13 smithi184 ceph-mon[102844]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:15 smithi196 ceph-mon[76349]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:15 smithi184 ceph-mon[100622]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:15 smithi184 ceph-mon[102844]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:17 smithi184 ceph-mon[100622]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:17.396 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:17 smithi184 ceph-mon[102844]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:17.397 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:17 smithi196 ceph-mon[76349]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:19 smithi196 ceph-mon[76349]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:19 smithi184 ceph-mon[100622]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:19 smithi184 ceph-mon[102844]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:21.181 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:49:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:21 smithi196 ceph-mon[76349]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:21.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:21 smithi184 ceph-mon[100622]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:21.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:21 smithi184 ceph-mon[102844]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 7h - - 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 18s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 19s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:49:21.715 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 19s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 19s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 18s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 19s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 19s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 19s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 7h 78.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:49:21.716 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:49:21.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:49:21.717 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:49:22.059 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:49:22.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:49:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:49:22.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:49:22.061 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:49:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-mon[100622]: from='client.82334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:22.182 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/741280718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:22.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-mon[102844]: from='client.82334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:22.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/741280718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:22.182 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:21.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:22.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:21 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:21.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:22.183 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:22.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:22.183 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:49:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:49:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:49:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:49:22.373 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:49:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:22 smithi196 ceph-mon[76349]: from='client.82334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/741280718' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:22.733 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:49:22.733 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:22.733 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:22.733 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:22.733 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='client.72381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2100532456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='client.72381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.403 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:23 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2100532456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='client.72381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:23 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2100532456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[100622]: from='client.72393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[102844]: from='client.72393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:24 smithi196 ceph-mon[76349]: from='client.72393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:25 smithi184 ceph-mon[100622]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:25 smithi184 ceph-mon[102844]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:25 smithi196 ceph-mon[76349]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:27 smithi184 ceph-mon[100622]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:27 smithi184 ceph-mon[102844]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:27 smithi196 ceph-mon[76349]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:28.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:29.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:29 smithi196 ceph-mon[76349]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:29 smithi184 ceph-mon[100622]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:29 smithi184 ceph-mon[102844]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:30.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:31.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:31 smithi196 ceph-mon[76349]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:31 smithi184 ceph-mon[100622]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:31 smithi184 ceph-mon[102844]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:49:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:32.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:32.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:32.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:33.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:33 smithi184 ceph-mon[100622]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:33.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:33 smithi184 ceph-mon[102844]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:33 smithi196 ceph-mon[76349]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:35 smithi184 ceph-mon[100622]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:35 smithi184 ceph-mon[102844]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:35 smithi196 ceph-mon[76349]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:36 smithi184 ceph-mon[100622]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:36 smithi184 ceph-mon[102844]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:36 smithi196 ceph-mon[76349]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:37.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:37 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:37 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:37 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:38 smithi184 ceph-mon[102844]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:38 smithi184 ceph-mon[100622]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:38 smithi196 ceph-mon[76349]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:41.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:41 smithi196 ceph-mon[76349]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:41 smithi184 ceph-mon[100622]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:41 smithi184 ceph-mon[102844]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:49:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:42.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:42.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:42.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[100622]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[102844]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:43 smithi196 ceph-mon[76349]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:43.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[102844]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[100622]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:44 smithi196 ceph-mon[76349]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:49:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:47 smithi184 ceph-mon[100622]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:47 smithi184 ceph-mon[102844]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:47 smithi196 ceph-mon[76349]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:48.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:49 smithi184 ceph-mon[100622]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:49 smithi184 ceph-mon[102844]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:49 smithi196 ceph-mon[76349]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:51 smithi184 ceph-mon[100622]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:51 smithi184 ceph-mon[102844]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:51 smithi196 ceph-mon[76349]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:49:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:49:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:52.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:52.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:49:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:49:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:49:52.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:49:53.029 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:49:53.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:53 smithi184 ceph-mon[100622]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:53 smithi184 ceph-mon[102844]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:53 smithi196 ceph-mon[76349]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 7h - - 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 7h - - 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 11s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 9s ago 7h 95.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 6h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:49:53.555 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 9s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 9s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 11s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 9s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 9s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 9s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 9s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 9s ago 7h 73.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:49:53.556 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:49:53.557 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:49:53.557 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 6h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:49:53.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:49:53.866 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:49:53.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:49:53.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:49:53.868 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:49:54.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:54 smithi184 ceph-mon[100622]: from='client.82367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:54.152 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4044113822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:54.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:54 smithi184 ceph-mon[102844]: from='client.82367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:54.153 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4044113822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:49:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:49:54.154 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:49:54.154 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:49:54.154 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:49:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:54 smithi196 ceph-mon[76349]: from='client.82367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4044113822' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:54.469 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:49:54.469 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:54.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:54.469 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:54.469 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:55 smithi196 ceph-mon[76349]: from='client.82373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:55 smithi196 ceph-mon[76349]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:55 smithi196 ceph-mon[76349]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:55 smithi196 ceph-mon[76349]: from='client.82391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3961473478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[100622]: from='client.82373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[100622]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[100622]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[100622]: from='client.82391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3961473478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[102844]: from='client.82373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[102844]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[102844]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[102844]: from='client.82391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:55.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3961473478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:57 smithi184 ceph-mon[100622]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:57 smithi184 ceph-mon[102844]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:57 smithi196 ceph-mon[76349]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:49:59 smithi196 ceph-mon[76349]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:49:59 smithi184 ceph-mon[100622]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:49:59 smithi184 ceph-mon[102844]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:49:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:01 smithi196 ceph-mon[76349]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:01 smithi184 ceph-mon[100622]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:01 smithi184 ceph-mon[102844]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:02.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:02.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:02.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:50:03.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[100622]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:03.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.107 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.108 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[102844]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:03 smithi196 ceph-mon[76349]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[102844]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[100622]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:05 smithi196 ceph-mon[76349]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:07 smithi184 ceph-mon[102844]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:07 smithi184 ceph-mon[100622]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:07 smithi196 ceph-mon[76349]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:09 smithi196 ceph-mon[76349]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:09 smithi184 ceph-mon[100622]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:09 smithi184 ceph-mon[102844]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:11.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:11 smithi196 ceph-mon[76349]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:11 smithi184 ceph-mon[100622]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:11 smithi184 ceph-mon[102844]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:50:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:12.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:12.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:12.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:13 smithi196 ceph-mon[76349]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:13 smithi184 ceph-mon[100622]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:13 smithi184 ceph-mon[102844]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:15 smithi196 ceph-mon[76349]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:15 smithi184 ceph-mon[100622]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:15 smithi184 ceph-mon[102844]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:17 smithi184 ceph-mon[100622]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:17.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:17 smithi184 ceph-mon[102844]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:17 smithi196 ceph-mon[76349]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:19 smithi196 ceph-mon[76349]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:19 smithi184 ceph-mon[100622]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:19 smithi184 ceph-mon[102844]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:21 smithi196 ceph-mon[76349]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:21 smithi184 ceph-mon[100622]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:21 smithi184 ceph-mon[102844]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:50:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:22.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:22.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:22.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:23.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[102844]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:23.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[100622]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:23.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:23 smithi196 ceph-mon[76349]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:23.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:24.777 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:50:25.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[100622]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:25.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.358 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[102844]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.374 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:50:25.374 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 7h - - 2024-09-21T02:50:25.374 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 7h - - 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 2s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 1s ago 7h 96.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 2s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 2s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 2s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 2s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 2s ago 7h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 1s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 2s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 2s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 2s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:50:25.375 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 7h 76.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:50:25.376 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:50:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:25 smithi196 ceph-mon[76349]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:25.701 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:50:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:50:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:50:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:50:25.702 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:50:25.984 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:50:25.985 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[100622]: from='client.82403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[100622]: from='client.82409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2853555325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[102844]: from='client.82403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[102844]: from='client.82409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2853555325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:26.306 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:26.306 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:26.306 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:26 smithi196 ceph-mon[76349]: from='client.82403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:26 smithi196 ceph-mon[76349]: from='client.82409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2853555325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[102844]: from='client.82415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[102844]: from='client.72465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[102844]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/361974994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[100622]: from='client.82415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[100622]: from='client.72465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[100622]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:27.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/361974994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:27 smithi196 ceph-mon[76349]: from='client.82415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:27 smithi196 ceph-mon[76349]: from='client.72465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:27 smithi196 ceph-mon[76349]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/361974994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:28.540 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:29 smithi184 ceph-mon[100622]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:29 smithi184 ceph-mon[102844]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:29 smithi196 ceph-mon[76349]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:31.421 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:31 smithi196 ceph-mon[76349]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:31 smithi184 ceph-mon[100622]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:31 smithi184 ceph-mon[102844]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:50:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:32.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:32.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:32.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:33 smithi196 ceph-mon[76349]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:33 smithi184 ceph-mon[100622]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:33 smithi184 ceph-mon[102844]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:35 smithi184 ceph-mon[100622]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:35 smithi184 ceph-mon[102844]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:35 smithi196 ceph-mon[76349]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:37 smithi184 ceph-mon[100622]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:37 smithi184 ceph-mon[102844]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:37 smithi196 ceph-mon[76349]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:39 smithi184 ceph-mon[100622]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:39 smithi184 ceph-mon[102844]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:39 smithi196 ceph-mon[76349]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:41 smithi184 ceph-mon[100622]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:41 smithi184 ceph-mon[102844]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:41 smithi196 ceph-mon[76349]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:42.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:42.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:42.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:50:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[100622]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[102844]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:43 smithi196 ceph-mon[76349]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:45.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[100622]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:45.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[102844]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:45.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:45 smithi196 ceph-mon[76349]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:50:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:46 smithi184 ceph-mon[100622]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:46 smithi184 ceph-mon[102844]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:46 smithi196 ceph-mon[76349]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:48 smithi184 ceph-mon[100622]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:48 smithi184 ceph-mon[102844]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:48 smithi196 ceph-mon[76349]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:51 smithi196 ceph-mon[76349]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:51 smithi184 ceph-mon[100622]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:51 smithi184 ceph-mon[102844]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:50:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:50:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:52.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:52.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:50:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:50:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:50:52.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:50:53.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:53 smithi196 ceph-mon[76349]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:53 smithi184 ceph-mon[100622]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:53 smithi184 ceph-mon[102844]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:55.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:55 smithi196 ceph-mon[76349]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:55 smithi184 ceph-mon[100622]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:55 smithi184 ceph-mon[102844]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:56.601 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:50:57.130 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 7h - - 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 7h - - 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 14s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 12s ago 7h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 14s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 12s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 14s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 14s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 12s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 14s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 14s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 12s ago 7h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:50:57.131 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 14s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 14s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 14s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 14s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 12s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 12s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 12s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 12s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 12s ago 7h 76.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:50:57.132 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 12s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:50:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:57 smithi196 ceph-mon[76349]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:57.441 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:57 smithi184 ceph-mon[100622]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:57.441 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:57 smithi184 ceph-mon[102844]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:57.441 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:50:57.441 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:50:57.442 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:50:57.710 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:50:57.711 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:50:57.711 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:50:57.711 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:50:57.711 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:50:57.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:50:58.021 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:58.021 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:58.021 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:58.021 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:58.021 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.72477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.72483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.82451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/379271769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1789573152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.72477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.72483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.82451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/379271769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1789573152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.72477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.72483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.82451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/379271769' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1789573152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:59 smithi196 ceph-mon[76349]: from='client.72501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:59.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:50:59 smithi196 ceph-mon[76349]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:59 smithi184 ceph-mon[100622]: from='client.72501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:50:59 smithi184 ceph-mon[100622]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:59 smithi184 ceph-mon[102844]: from='client.72501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:50:59 smithi184 ceph-mon[102844]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:01 smithi196 ceph-mon[76349]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:01 smithi184 ceph-mon[100622]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:01 smithi184 ceph-mon[102844]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:02] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:51:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:02.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:02.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:02.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:03 smithi196 ceph-mon[76349]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:03 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[102844]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:03.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[100622]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:03.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:03.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:03 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[100622]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[102844]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:05 smithi196 ceph-mon[76349]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:05.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:07.293 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:07 smithi184 ceph-mon[100622]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:07.293 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:07 smithi184 ceph-mon[102844]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:07 smithi196 ceph-mon[76349]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:09 smithi196 ceph-mon[76349]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:09 smithi184 ceph-mon[100622]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:09 smithi184 ceph-mon[102844]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:11 smithi196 ceph-mon[76349]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:11 smithi184 ceph-mon[100622]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:11 smithi184 ceph-mon[102844]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:12] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:51:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:12.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:12.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:12.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:13 smithi196 ceph-mon[76349]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:13 smithi184 ceph-mon[100622]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:13 smithi184 ceph-mon[102844]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:15 smithi196 ceph-mon[76349]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:15 smithi184 ceph-mon[100622]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:15 smithi184 ceph-mon[102844]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:17 smithi196 ceph-mon[76349]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:17 smithi184 ceph-mon[100622]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:17 smithi184 ceph-mon[102844]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:19 smithi196 ceph-mon[76349]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:19 smithi184 ceph-mon[100622]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:19 smithi184 ceph-mon[102844]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:21 smithi196 ceph-mon[76349]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:21 smithi184 ceph-mon[100622]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:21 smithi184 ceph-mon[102844]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:51:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:22.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:22.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:22.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[102844]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[100622]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:23 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:23 smithi196 ceph-mon[76349]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:23 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:25 smithi196 ceph-mon[76349]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[102844]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:25.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.423 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[100622]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:25.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:25.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:27.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:27 smithi196 ceph-mon[76349]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:27.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:27 smithi184 ceph-mon[100622]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:27.395 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:27 smithi184 ceph-mon[102844]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:28.316 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:51:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:28.842 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:51:28.842 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 7h - - 2024-09-21T02:51:28.842 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 7h - - 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 5s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 4s ago 7h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 5s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 5s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 5s ago 7h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 5s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 5s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 4s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 5s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 5s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 5s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:51:28.843 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 4s ago 7h 77.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:51:28.844 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:51:29.154 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:51:29.154 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:51:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:51:29.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:51:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:29 smithi196 ceph-mon[76349]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:51:29.428 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:51:29.429 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:29 smithi184 ceph-mon[100622]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:29.429 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:29 smithi184 ceph-mon[102844]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:29.738 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:51:29.739 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:29.739 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:29.739 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:29.739 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='client.82475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='client.82481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='client.72525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2295775242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1077458887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='client.82475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='client.82481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='client.72525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2295775242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1077458887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='client.82475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='client.82481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='client.72525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2295775242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1077458887' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:31 smithi184 ceph-mon[102844]: from='client.72537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:31 smithi184 ceph-mon[102844]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:31 smithi184 ceph-mon[100622]: from='client.72537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:31 smithi184 ceph-mon[100622]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:31 smithi196 ceph-mon[76349]: from='client.72537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:31 smithi196 ceph-mon[76349]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:51:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:32.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:32.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:32.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:33.376 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:33 smithi196 ceph-mon[76349]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:33.481 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:33 smithi184 ceph-mon[100622]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:33.481 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:33 smithi184 ceph-mon[102844]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:35.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:35 smithi196 ceph-mon[76349]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:35 smithi184 ceph-mon[100622]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:35 smithi184 ceph-mon[102844]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:37 smithi184 ceph-mon[100622]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:37 smithi184 ceph-mon[102844]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:37 smithi196 ceph-mon[76349]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:39 smithi184 ceph-mon[100622]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:39 smithi184 ceph-mon[102844]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:39 smithi196 ceph-mon[76349]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:41 smithi184 ceph-mon[100622]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:41 smithi184 ceph-mon[102844]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:41 smithi196 ceph-mon[76349]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:51:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:42.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:42.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:42.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[100622]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[102844]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:43 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:43 smithi196 ceph-mon[76349]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:43 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[100622]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[102844]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:45 smithi196 ceph-mon[76349]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:51:47.349 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:47 smithi184 ceph-mon[100622]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:47.349 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:47 smithi184 ceph-mon[102844]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:47 smithi196 ceph-mon[76349]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:48.490 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:48.490 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:49 smithi184 ceph-mon[102844]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:49 smithi184 ceph-mon[100622]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:49 smithi196 ceph-mon[76349]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:51 smithi184 ceph-mon[102844]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:51 smithi184 ceph-mon[100622]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:51 smithi196 ceph-mon[76349]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:51:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:51:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:52.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:52.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:51:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:51:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:51:52.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:51:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:53 smithi184 ceph-mon[100622]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:53 smithi184 ceph-mon[102844]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:53 smithi196 ceph-mon[76349]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:55 smithi184 ceph-mon[100622]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:55 smithi184 ceph-mon[102844]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:55 smithi196 ceph-mon[76349]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:57 smithi184 ceph-mon[100622]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:57 smithi184 ceph-mon[102844]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:57 smithi196 ceph-mon[76349]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:58.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:59 smithi184 ceph-mon[100622]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:51:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:59 smithi184 ceph-mon[102844]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:51:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:59.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:59 smithi196 ceph-mon[76349]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:59.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:51:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:00.037 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:52:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:52:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 7h - - 2024-09-21T02:52:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 7h - - 2024-09-21T02:52:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 17s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:52:00.564 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 15s ago 7h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 17s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 15s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 17s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 17s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 15s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 17s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 17s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 15s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 17s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 17s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 17s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 17s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:52:00.565 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 15s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 15s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 15s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 15s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 15s ago 7h 77.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:52:00.566 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 15s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:52:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:52:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:52:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:52:00.876 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:52:01.144 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:52:01.145 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:52:01.457 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:52:01.457 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:01.457 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:01.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[102844]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:01.457 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[102844]: from='client.72549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:01.458 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/431752431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:01.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[100622]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:01.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[100622]: from='client.72549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:01.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/431752431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:01 smithi196 ceph-mon[76349]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:01 smithi196 ceph-mon[76349]: from='client.72549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/431752431' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[102844]: from='client.72555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[102844]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[102844]: from='client.82535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2536965675' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[100622]: from='client.72555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[100622]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[100622]: from='client.82535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2536965675' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:02.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:02.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:02.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:02 smithi196 ceph-mon[76349]: from='client.72555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:02 smithi196 ceph-mon[76349]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:02 smithi196 ceph-mon[76349]: from='client.82535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2536965675' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:03.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:03 smithi184 ceph-mon[100622]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:03.487 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:03 smithi184 ceph-mon[102844]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:03 smithi196 ceph-mon[76349]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[100622]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:04 smithi184 ceph-mon[102844]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:04 smithi196 ceph-mon[76349]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:07.234 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:06 smithi184 ceph-mon[100622]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:07.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:06 smithi184 ceph-mon[102844]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:06 smithi196 ceph-mon[76349]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:08.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:08.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:08 smithi184 ceph-mon[100622]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:08 smithi184 ceph-mon[102844]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:08 smithi196 ceph-mon[76349]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:11 smithi196 ceph-mon[76349]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:11 smithi184 ceph-mon[100622]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:11 smithi184 ceph-mon[102844]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:52:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:12.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:12.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:12.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:13 smithi184 ceph-mon[102844]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:13.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:13 smithi184 ceph-mon[100622]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:13 smithi196 ceph-mon[76349]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:15.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:15 smithi196 ceph-mon[76349]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:15.346 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:15 smithi184 ceph-mon[100622]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:15 smithi184 ceph-mon[102844]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:17 smithi196 ceph-mon[76349]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:17 smithi184 ceph-mon[100622]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:17 smithi184 ceph-mon[102844]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:19 smithi196 ceph-mon[76349]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:19 smithi184 ceph-mon[100622]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:19 smithi184 ceph-mon[102844]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:21 smithi196 ceph-mon[76349]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:21 smithi184 ceph-mon[102844]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:21 smithi184 ceph-mon[100622]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:52:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:22.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:22.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:22.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:23.373 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:23 smithi184 ceph-mon[100622]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:23.373 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:23 smithi184 ceph-mon[102844]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:23 smithi196 ceph-mon[76349]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:24 smithi196 ceph-mon[76349]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[100622]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:24 smithi184 ceph-mon[102844]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:27 smithi196 ceph-mon[76349]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:27.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:27 smithi184 ceph-mon[100622]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:27.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:27 smithi184 ceph-mon[102844]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:28.295 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:28.295 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:29.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:29 smithi196 ceph-mon[76349]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:29.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:29 smithi184 ceph-mon[100622]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:29.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:29 smithi184 ceph-mon[102844]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:31 smithi184 ceph-mon[100622]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:31 smithi184 ceph-mon[102844]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:31 smithi196 ceph-mon[76349]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:31.750 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:52:32.280 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:52:32.280 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 7h - - 2024-09-21T02:52:32.280 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 7h - - 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 8s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 7s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 8s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 7s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 8s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 8s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 8s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 8s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 7s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 8s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 8s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 8s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:52:32.281 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 7s ago 7h 78.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:52:32.282 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:52:32.282 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:52:32.282 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:32.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:32.283 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:32.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:32.283 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:32.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:32.591 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:52:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:52:32.859 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:52:32.859 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:52:32.860 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:52:33.177 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:52:33.177 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:33.177 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:33.177 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:33.177 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[100622]: from='client.82547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.177 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[100622]: from='client.82553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[100622]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3854680289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[102844]: from='client.82547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[102844]: from='client.82553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[102844]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:33.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3854680289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:33 smithi196 ceph-mon[76349]: from='client.82547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:33 smithi196 ceph-mon[76349]: from='client.82553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:33 smithi196 ceph-mon[76349]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3854680289' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:34 smithi196 ceph-mon[76349]: from='client.72597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:34 smithi196 ceph-mon[76349]: from='client.82571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2982507107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[100622]: from='client.72597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[100622]: from='client.82571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2982507107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[102844]: from='client.72597 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[102844]: from='client.82571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2982507107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:35.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:35 smithi196 ceph-mon[76349]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:35 smithi184 ceph-mon[100622]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:35 smithi184 ceph-mon[102844]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:37.382 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:37 smithi196 ceph-mon[76349]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:37 smithi184 ceph-mon[100622]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:37 smithi184 ceph-mon[102844]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:39.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:39 smithi196 ceph-mon[76349]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:39 smithi184 ceph-mon[100622]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:39 smithi184 ceph-mon[102844]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:41 smithi196 ceph-mon[76349]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:41 smithi184 ceph-mon[100622]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:41 smithi184 ceph-mon[102844]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:52:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:42.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:42.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:42.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:43 smithi196 ceph-mon[76349]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:43.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:43 smithi184 ceph-mon[100622]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:43.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:43 smithi184 ceph-mon[102844]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[102844]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[100622]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:44 smithi196 ceph-mon[76349]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:52:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:47 smithi184 ceph-mon[100622]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:47 smithi184 ceph-mon[102844]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:47 smithi196 ceph-mon[76349]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:48.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:48.460 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:49 smithi184 ceph-mon[100622]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:49 smithi184 ceph-mon[102844]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:49 smithi196 ceph-mon[76349]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:51 smithi184 ceph-mon[100622]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:51 smithi184 ceph-mon[102844]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:51 smithi196 ceph-mon[76349]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:52:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:52:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:52.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:52.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:52:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:52:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:52:52.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:52:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:53 smithi184 ceph-mon[100622]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:53 smithi184 ceph-mon[102844]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:53 smithi196 ceph-mon[76349]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:55.513 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:55 smithi196 ceph-mon[76349]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:55 smithi184 ceph-mon[100622]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:55 smithi184 ceph-mon[102844]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:57.395 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:57 smithi196 ceph-mon[76349]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:57 smithi184 ceph-mon[100622]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:57 smithi184 ceph-mon[102844]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:58.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:52:59 smithi184 ceph-mon[100622]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:52:59 smithi184 ceph-mon[102844]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:52:59 smithi196 ceph-mon[76349]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:01 smithi184 ceph-mon[102844]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:01 smithi184 ceph-mon[100622]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:01 smithi196 ceph-mon[76349]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:53:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:02.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:02.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:02.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:03.474 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:03 smithi184 ceph-mon[100622]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:03.474 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:03 smithi184 ceph-mon[102844]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:03.474 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:53:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:03 smithi196 ceph-mon[76349]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 7h - - 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 7h - - 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 0s ago 7h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 18s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 0s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 18s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 0s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:53:04.018 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 0s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 18s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 0s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 0s ago 7h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 18s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 18s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 18s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 18s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 18s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 18s ago 7h 78.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:53:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:53:04.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:53:04.020 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 18s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:53:04.326 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:53:04.327 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:53:04.594 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:53:04.594 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:53:04.594 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:53:04.595 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:53:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='client.82583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='client.72627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='client.82595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:04.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1001071886' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:04.905 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:53:04.905 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:04.905 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:04.905 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:04.905 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:04.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='client.82583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.905 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='client.72627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='client.82595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1001071886' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='client.82583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='client.72627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='client.82595 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:04.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:04.907 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1001071886' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:05 smithi196 ceph-mon[76349]: from='client.82607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2089430880' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:05.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[100622]: from='client.82607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2089430880' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[102844]: from='client.82607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2089430880' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:06 smithi196 ceph-mon[76349]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:06 smithi184 ceph-mon[100622]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:06 smithi184 ceph-mon[102844]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:07 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:07 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:08.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:07 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:08 smithi196 ceph-mon[76349]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:08 smithi184 ceph-mon[100622]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.924 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:08 smithi184 ceph-mon[102844]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:11.341 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:11 smithi184 ceph-mon[100622]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:11.341 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:11 smithi184 ceph-mon[102844]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:11 smithi196 ceph-mon[76349]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:53:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:12.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:12.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:12.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:13 smithi184 ceph-mon[102844]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:13 smithi184 ceph-mon[100622]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:13 smithi196 ceph-mon[76349]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:15 smithi196 ceph-mon[76349]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:15 smithi184 ceph-mon[102844]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:15 smithi184 ceph-mon[100622]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:17.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:17 smithi196 ceph-mon[76349]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:17 smithi184 ceph-mon[100622]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:17 smithi184 ceph-mon[102844]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:18.366 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:19 smithi196 ceph-mon[76349]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:19 smithi184 ceph-mon[100622]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:19 smithi184 ceph-mon[102844]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:21.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:21 smithi196 ceph-mon[76349]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:21 smithi184 ceph-mon[100622]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:21 smithi184 ceph-mon[102844]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:53:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:22.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:22.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:22.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:23.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:23 smithi184 ceph-mon[100622]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:23.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:23 smithi184 ceph-mon[102844]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:23.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:23 smithi196 ceph-mon[76349]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[102844]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:24 smithi184 ceph-mon[100622]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:25.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:24 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:24 smithi196 ceph-mon[76349]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[100622]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:26 smithi184 ceph-mon[102844]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:26 smithi196 ceph-mon[76349]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:27 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:27 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:27 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:28.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:28 smithi196 ceph-mon[76349]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:28 smithi184 ceph-mon[100622]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:28 smithi184 ceph-mon[102844]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:29.687 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:29.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:29.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:30 smithi196 ceph-mon[76349]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:30.706 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:30 smithi184 ceph-mon[100622]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:30.706 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:30 smithi184 ceph-mon[102844]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:32.162 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:53:32.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:32.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:32.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:32.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:32.162 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:32.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:33 smithi196 ceph-mon[76349]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:33.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:33 smithi184 ceph-mon[100622]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:33.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:33 smithi184 ceph-mon[102844]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:35.201 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:53:35.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:35 smithi196 ceph-mon[76349]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:35 smithi184 ceph-mon[100622]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:35 smithi184 ceph-mon[102844]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 7h - - 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 7h - - 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 11s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 10s ago 7h 92.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 11s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 10s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 11s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 11s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 10s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 11s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 11s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:53:35.728 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 10s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 11s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 11s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 11s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 10s ago 7h 78.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:53:35.729 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:53:36.042 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:53:36.042 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:53:36.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:53:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:36 smithi184 ceph-mon[100622]: from='client.82619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2318828389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:36 smithi184 ceph-mon[102844]: from='client.82619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2318828389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:53:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:53:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:53:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:53:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:53:36.314 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:53:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:36 smithi196 ceph-mon[76349]: from='client.82619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2318828389' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:36.623 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:53:36.624 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:36.624 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:36.624 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:36.624 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:37 smithi196 ceph-mon[76349]: from='client.72663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:37 smithi196 ceph-mon[76349]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:37 smithi196 ceph-mon[76349]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:37.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3962144391' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[100622]: from='client.72663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[100622]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[100622]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3962144391' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[102844]: from='client.72663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[102844]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[102844]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3962144391' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:38 smithi196 ceph-mon[76349]: from='client.72681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:38 smithi184 ceph-mon[100622]: from='client.72681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:38 smithi184 ceph-mon[102844]: from='client.72681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:39.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:39 smithi196 ceph-mon[76349]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:39 smithi184 ceph-mon[100622]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:39 smithi184 ceph-mon[102844]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:41 smithi196 ceph-mon[76349]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:41 smithi184 ceph-mon[100622]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:41 smithi184 ceph-mon[102844]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:53:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:42.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:42.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:42.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:43 smithi196 ceph-mon[76349]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:43 smithi184 ceph-mon[100622]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:43 smithi184 ceph-mon[102844]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:44 smithi196 ceph-mon[76349]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[102844]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[100622]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:45.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[100622]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:46 smithi184 ceph-mon[102844]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:53:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:46 smithi196 ceph-mon[76349]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:47 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:47 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:47 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:48 smithi184 ceph-mon[100622]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:48 smithi184 ceph-mon[102844]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:48 smithi196 ceph-mon[76349]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:51 smithi184 ceph-mon[100622]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:51 smithi184 ceph-mon[102844]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:51 smithi196 ceph-mon[76349]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:53:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:53:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:52.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:52.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:53:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:53:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:53:52.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:53:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:53 smithi196 ceph-mon[76349]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:53 smithi184 ceph-mon[102844]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:53 smithi184 ceph-mon[100622]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:55.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:55 smithi196 ceph-mon[76349]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:55 smithi184 ceph-mon[100622]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:55 smithi184 ceph-mon[102844]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:57.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:57 smithi196 ceph-mon[76349]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:57 smithi184 ceph-mon[100622]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:57 smithi184 ceph-mon[102844]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:58.359 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:59.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:53:59 smithi196 ceph-mon[76349]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:53:59 smithi184 ceph-mon[100622]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:53:59 smithi184 ceph-mon[102844]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:00.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:01 smithi196 ceph-mon[76349]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:01 smithi184 ceph-mon[100622]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:01 smithi184 ceph-mon[102844]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:54:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:02.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:02.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:02.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:03 smithi184 ceph-mon[100622]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:03 smithi184 ceph-mon[102844]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:03 smithi196 ceph-mon[76349]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[100622]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:05 smithi184 ceph-mon[102844]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:05.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:05 smithi196 ceph-mon[76349]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:06.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.920 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[100622]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:06.920 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:06.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:06 smithi184 ceph-mon[102844]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:06.921 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:54:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:07.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:07.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:06 smithi196 ceph-mon[76349]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:07.448 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:54:07.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 7h - - 2024-09-21T02:54:07.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 7h - - 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 3s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 1s ago 7h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 3s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 1s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (6h) 3s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 3s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 1s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 3s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 3s ago 7h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 1s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 3s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 3s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 3s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 3s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:54:07.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 1s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 1s ago 7h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 1s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 1s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:54:07.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 1s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:54:07.765 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:54:07.765 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:54:07.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:54:07.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:54:07.767 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:54:08.040 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:54:08.040 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:54:08.040 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:54:08.040 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:54:08.040 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:54:08.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:54:08.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:07 smithi184 ceph-mon[100622]: from='client.72693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:07 smithi184 ceph-mon[100622]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:07 smithi184 ceph-mon[102844]: from='client.72693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.042 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:07 smithi184 ceph-mon[102844]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:07 smithi196 ceph-mon[76349]: from='client.72693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:07 smithi196 ceph-mon[76349]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:08.351 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:54:08.351 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:08.351 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:08.351 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:08.352 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4207324728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: from='client.82679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:54:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2197222835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4207324728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: from='client.82679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:54:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2197222835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4207324728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: from='client.82679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T02:54:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2197222835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:11.245 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:11 smithi184 ceph-mon[100622]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:11.245 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:11 smithi184 ceph-mon[102844]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:11 smithi196 ceph-mon[76349]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:54:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:12.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:12.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:12.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:13.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:13 smithi196 ceph-mon[76349]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:13 smithi184 ceph-mon[100622]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:13 smithi184 ceph-mon[102844]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:54:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:15 smithi196 ceph-mon[76349]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:54:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:15 smithi184 ceph-mon[100622]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:54:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:15 smithi184 ceph-mon[102844]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:54:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:17.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:17 smithi196 ceph-mon[76349]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:54:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:17 smithi184 ceph-mon[100622]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:54:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:17 smithi184 ceph-mon[102844]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T02:54:19.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:19.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:19 smithi196 ceph-mon[76349]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:54:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:19 smithi184 ceph-mon[100622]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:19 smithi184 ceph-mon[102844]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T02:54:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:21 smithi184 ceph-mon[100622]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T02:54:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:21 smithi184 ceph-mon[102844]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T02:54:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:21 smithi196 ceph-mon[76349]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-21T02:54:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:54:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:22.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:22.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:22.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:23 smithi184 ceph-mon[100622]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T02:54:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:23 smithi184 ceph-mon[102844]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T02:54:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:23 smithi196 ceph-mon[76349]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T02:54:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:25 smithi196 ceph-mon[76349]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:54:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[100622]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:54:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[102844]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T02:54:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[100622]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:26 smithi184 ceph-mon[102844]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:27.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:26 smithi196 ceph-mon[76349]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:29 smithi196 ceph-mon[76349]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:54:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:29 smithi184 ceph-mon[100622]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:54:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:29 smithi184 ceph-mon[102844]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T02:54:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:31 smithi196 ceph-mon[76349]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:54:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:31 smithi184 ceph-mon[100622]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:54:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:31 smithi184 ceph-mon[102844]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:54:32.114 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:54:32.114 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:32.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:32.114 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:32.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:32.114 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:32.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:33.266 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:33 smithi184 ceph-mon[100622]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:54:33.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:33 smithi184 ceph-mon[102844]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:54:33.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:33 smithi196 ceph-mon[76349]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:54:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:35 smithi184 ceph-mon[100622]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:54:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:35 smithi184 ceph-mon[102844]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:54:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:35 smithi196 ceph-mon[76349]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:54:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:37 smithi184 ceph-mon[100622]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:54:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:37 smithi184 ceph-mon[102844]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:54:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:37 smithi196 ceph-mon[76349]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:54:38.638 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 7h - - 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 7h - - 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 15s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 13s ago 7h 94.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (6h) 15s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 13s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 15s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 15s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 13s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 15s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:54:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 15s ago 7h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 13s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 15s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 13s ago 7h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 13s ago 7h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 13s ago 7h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 13s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (6h) 13s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:54:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 13s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:54:39.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:54:39.477 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:54:39.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:39.477 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:39 smithi184 ceph-mon[100622]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:54:39.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:39.477 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:39 smithi184 ceph-mon[102844]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:54:39.612 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:39.613 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:39 smithi196 ceph-mon[76349]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T02:54:39.745 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:54:39.746 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:40.059 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:40 smithi196 ceph-mon[76349]: from='client.82691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:40 smithi196 ceph-mon[76349]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:40 smithi196 ceph-mon[76349]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3162392195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1420534300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[102844]: from='client.82691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[102844]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[102844]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3162392195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1420534300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[100622]: from='client.82691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[100622]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[100622]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3162392195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1420534300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:41 smithi184 ceph-mon[100622]: from='client.72753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:41 smithi184 ceph-mon[100622]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T02:54:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:41 smithi184 ceph-mon[102844]: from='client.72753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:41 smithi184 ceph-mon[102844]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T02:54:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:41 smithi196 ceph-mon[76349]: from='client.72753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:41 smithi196 ceph-mon[76349]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T02:54:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:54:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:42.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:42.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:42.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:43 smithi184 ceph-mon[102844]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:54:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:43 smithi184 ceph-mon[100622]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:54:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:43 smithi196 ceph-mon[76349]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[102844]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[100622]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:54:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:45 smithi196 ceph-mon[76349]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-21T02:54:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.269 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.275 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[102844]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:47.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.276 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:46 smithi184 ceph-mon[100622]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:54:47.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:46 smithi196 ceph-mon[76349]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-21T02:54:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:49 smithi196 ceph-mon[76349]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:54:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:49 smithi184 ceph-mon[100622]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:54:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:49 smithi184 ceph-mon[102844]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:54:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:51 smithi196 ceph-mon[76349]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:54:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:51 smithi184 ceph-mon[100622]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:54:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:51 smithi184 ceph-mon[102844]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 47 op/s 2024-09-21T02:54:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:54:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:54:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:52.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:52.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:54:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:54:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:54:52.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:54:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:53 smithi184 ceph-mon[100622]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:54:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:53 smithi184 ceph-mon[102844]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:54:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:53 smithi196 ceph-mon[76349]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:54:55.267 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:55 smithi184 ceph-mon[100622]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:55.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:55 smithi184 ceph-mon[102844]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:55 smithi196 ceph-mon[76349]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:57 smithi184 ceph-mon[100622]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:54:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:57 smithi184 ceph-mon[102844]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:54:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:57 smithi196 ceph-mon[76349]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:54:59.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:59.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:54:59 smithi196 ceph-mon[76349]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:54:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:54:59 smithi184 ceph-mon[100622]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:54:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:54:59 smithi184 ceph-mon[102844]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:00.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:01 smithi184 ceph-mon[100622]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:01 smithi184 ceph-mon[102844]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:01.552 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:01 smithi196 ceph-mon[76349]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:55:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:02.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:02.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:02.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:03.488 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:03 smithi184 ceph-mon[100622]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:03.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:03 smithi184 ceph-mon[102844]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:03 smithi196 ceph-mon[76349]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[100622]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[102844]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:05 smithi196 ceph-mon[76349]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[100622]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.290 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:07 smithi184 ceph-mon[102844]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:07.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:07.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:07 smithi196 ceph-mon[76349]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:09 smithi184 ceph-mon[102844]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:09 smithi184 ceph-mon[100622]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:09 smithi196 ceph-mon[76349]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:10.364 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 7h - - 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 7h - - 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 6s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 4s ago 7h 96.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 6s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:55:10.888 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (6h) 4s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 6s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 6s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 4s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 6s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 6s ago 7h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 4s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 6s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 6s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 6s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 6s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 4s ago 7h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 4s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 4s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:55:10.889 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 4s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:55:10.890 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 4s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:55:10.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:55:10.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:55:10.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:55:10.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 4s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:11.208 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:55:11.209 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:55:11.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:55:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:55:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:55:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:55:11.478 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:55:11.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:11 smithi184 ceph-mon[100622]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:11.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:11 smithi184 ceph-mon[102844]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:11 smithi196 ceph-mon[76349]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:11.787 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:55:11.788 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:11.788 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:12.047 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:55:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:12.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:12.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:12.048 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:12.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[100622]: from='client.72765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[100622]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[100622]: from='client.82739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3643447162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/494840765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[102844]: from='client.72765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[102844]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[102844]: from='client.82739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3643447162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:12.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/494840765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:12 smithi196 ceph-mon[76349]: from='client.72765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:12 smithi196 ceph-mon[76349]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:12 smithi196 ceph-mon[76349]: from='client.82739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3643447162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/494840765' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:13 smithi184 ceph-mon[100622]: from='client.72789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:13 smithi184 ceph-mon[100622]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:13 smithi184 ceph-mon[102844]: from='client.72789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:13 smithi184 ceph-mon[102844]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:13 smithi196 ceph-mon[76349]: from='client.72789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:13 smithi196 ceph-mon[76349]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:15 smithi184 ceph-mon[100622]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:15 smithi184 ceph-mon[102844]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:15 smithi196 ceph-mon[76349]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:17 smithi184 ceph-mon[100622]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:17 smithi184 ceph-mon[102844]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:17 smithi196 ceph-mon[76349]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:19 smithi184 ceph-mon[100622]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:19 smithi184 ceph-mon[102844]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:19 smithi196 ceph-mon[76349]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:21.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:21 smithi196 ceph-mon[76349]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:21 smithi184 ceph-mon[100622]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:21 smithi184 ceph-mon[102844]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:55:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:22.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:22.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:22.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:23 smithi184 ceph-mon[100622]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:23 smithi184 ceph-mon[102844]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:23 smithi196 ceph-mon[76349]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[100622]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[102844]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:25 smithi196 ceph-mon[76349]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[100622]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[102844]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:27 smithi196 ceph-mon[76349]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:29.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:29.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:29 smithi196 ceph-mon[76349]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:29 smithi184 ceph-mon[100622]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:29 smithi184 ceph-mon[102844]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:30.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:31 smithi184 ceph-mon[100622]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:31 smithi184 ceph-mon[102844]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:31 smithi196 ceph-mon[76349]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:55:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:32.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:32.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:32.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:33.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:33 smithi184 ceph-mon[100622]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:33.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:33 smithi184 ceph-mon[102844]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:33 smithi196 ceph-mon[76349]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:35.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:35 smithi184 ceph-mon[100622]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:35.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:35 smithi184 ceph-mon[102844]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:35 smithi196 ceph-mon[76349]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:37 smithi184 ceph-mon[100622]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:37 smithi184 ceph-mon[102844]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:37 smithi196 ceph-mon[76349]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:39 smithi184 ceph-mon[100622]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:39 smithi184 ceph-mon[102844]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:39 smithi196 ceph-mon[76349]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:41 smithi184 ceph-mon[100622]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:41 smithi184 ceph-mon[102844]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:41.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:41 smithi196 ceph-mon[76349]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:42.083 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:55:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:55:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:42.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:42.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:42.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:42.612 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:55:42.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 7h - - 2024-09-21T02:55:42.612 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 7h - - 2024-09-21T02:55:42.612 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 18s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 16s ago 7h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 18s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 16s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 18s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (6h) 18s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 16s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 18s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 18s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 16s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 18s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 18s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:55:42.613 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 16s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 16s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 16s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 16s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 16s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:55:42.614 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 16s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:55:42.921 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:55:42.921 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:55:42.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:55:42.923 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:55:43.193 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:55:43.193 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:55:43.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:55:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:43 smithi196 ceph-mon[76349]: from='client.72801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:43 smithi196 ceph-mon[76349]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:43.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1559313438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:43.502 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:55:43.502 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:43.503 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:43.503 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:43.503 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[100622]: from='client.72801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[100622]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1559313438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[102844]: from='client.72801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[102844]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:43.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1559313438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[100622]: from='client.82769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[100622]: from='client.72813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[100622]: from='client.82787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3654808846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[102844]: from='client.82769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[102844]: from='client.72813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[102844]: from='client.82787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3654808846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:44 smithi196 ceph-mon[76349]: from='client.82769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:44 smithi196 ceph-mon[76349]: from='client.72813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:44 smithi196 ceph-mon[76349]: from='client.82787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3654808846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[100622]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[102844]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:45 smithi196 ceph-mon[76349]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[100622]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[102844]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:47 smithi196 ceph-mon[76349]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:55:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:49 smithi184 ceph-mon[100622]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:49 smithi184 ceph-mon[102844]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:49 smithi196 ceph-mon[76349]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:51 smithi184 ceph-mon[100622]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:51 smithi184 ceph-mon[102844]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:51 smithi196 ceph-mon[76349]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:55:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:55:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:52.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:52.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:55:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:55:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:55:52.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:55:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:53 smithi196 ceph-mon[76349]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:53 smithi184 ceph-mon[100622]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:53 smithi184 ceph-mon[102844]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:55.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:55 smithi184 ceph-mon[100622]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:55.424 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:55 smithi184 ceph-mon[102844]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:55 smithi196 ceph-mon[76349]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:57.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:57 smithi184 ceph-mon[100622]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:57.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:57 smithi184 ceph-mon[102844]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:57 smithi196 ceph-mon[76349]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[100622]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[102844]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:55:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:59 smithi196 ceph-mon[76349]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:55:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:01.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:01 smithi196 ceph-mon[76349]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:01 smithi184 ceph-mon[100622]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:01 smithi184 ceph-mon[102844]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:56:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:02.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:02.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:02.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:03 smithi196 ceph-mon[76349]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:03 smithi184 ceph-mon[100622]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:03 smithi184 ceph-mon[102844]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[100622]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[102844]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:05 smithi196 ceph-mon[76349]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:05.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[102844]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[100622]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:07 smithi196 ceph-mon[76349]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:09 smithi184 ceph-mon[102844]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:09 smithi184 ceph-mon[100622]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:09 smithi196 ceph-mon[76349]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:11 smithi184 ceph-mon[100622]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:11 smithi184 ceph-mon[102844]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:11 smithi196 ceph-mon[76349]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:56:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:12.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:12.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:12.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:13 smithi184 ceph-mon[100622]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:13 smithi184 ceph-mon[102844]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:13 smithi196 ceph-mon[76349]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:13.790 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:56:14.316 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 7h - - 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 7h - - 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 9s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 7s ago 7h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 9s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 7s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 9s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 9s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (6h) 7s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (6h) 9s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 9s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 7s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 9s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:56:14.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 9s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 9s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 9s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 7s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 7s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 7s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 7s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 7s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:56:14.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 7s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:56:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:14.626 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:56:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:56:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:56:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:56:14.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:56:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:14.895 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:56:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:56:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:56:14.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:56:15.207 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:56:15.207 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:15.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:15.207 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:15.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[100622]: from='client.72837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[100622]: from='client.72843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[100622]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4192647885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2989516259' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[102844]: from='client.72837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[102844]: from='client.72843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[102844]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4192647885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2989516259' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:15 smithi196 ceph-mon[76349]: from='client.72837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:15 smithi196 ceph-mon[76349]: from='client.72843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:15 smithi196 ceph-mon[76349]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4192647885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2989516259' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:16 smithi184 ceph-mon[100622]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:16 smithi184 ceph-mon[100622]: from='client.82823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:16 smithi184 ceph-mon[102844]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:16 smithi184 ceph-mon[102844]: from='client.82823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:16 smithi196 ceph-mon[76349]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:16.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:16 smithi196 ceph-mon[76349]: from='client.82823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:17.525 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:17 smithi184 ceph-mon[100622]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:17.525 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:17 smithi184 ceph-mon[102844]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:17 smithi196 ceph-mon[76349]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.463 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:18.463 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:18 smithi184 ceph-mon[100622]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:18.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:18 smithi184 ceph-mon[102844]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:18 smithi196 ceph-mon[76349]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:21.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:21 smithi184 ceph-mon[100622]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:21.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:21 smithi184 ceph-mon[102844]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:21 smithi196 ceph-mon[76349]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:56:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:22.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:22.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:22.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:23.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:23 smithi196 ceph-mon[76349]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:23 smithi184 ceph-mon[100622]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:23 smithi184 ceph-mon[102844]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[100622]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[102844]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:25 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:25 smithi196 ceph-mon[76349]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:25 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[100622]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[102844]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:27 smithi196 ceph-mon[76349]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:29.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:29.442 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:29 smithi196 ceph-mon[76349]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:29 smithi184 ceph-mon[100622]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:29 smithi184 ceph-mon[102844]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:31 smithi184 ceph-mon[100622]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:31 smithi184 ceph-mon[102844]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:31 smithi196 ceph-mon[76349]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:56:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:32.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:32.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:32.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:33 smithi184 ceph-mon[100622]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:33 smithi184 ceph-mon[102844]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:33 smithi196 ceph-mon[76349]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:35 smithi184 ceph-mon[100622]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:35 smithi184 ceph-mon[102844]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:35 smithi196 ceph-mon[76349]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:37 smithi184 ceph-mon[100622]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:37 smithi184 ceph-mon[102844]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:37 smithi196 ceph-mon[76349]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:39.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:39 smithi184 ceph-mon[100622]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:39.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:39.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:39 smithi184 ceph-mon[102844]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:39 smithi196 ceph-mon[76349]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:41 smithi184 ceph-mon[100622]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:41 smithi184 ceph-mon[102844]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:41 smithi196 ceph-mon[76349]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:56:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:42.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:42.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:42.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:43 smithi184 ceph-mon[100622]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:43 smithi184 ceph-mon[102844]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:43.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:43 smithi196 ceph-mon[76349]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[100622]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[102844]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.376 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:45 smithi196 ceph-mon[76349]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:45.494 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:56:46.021 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:56:46.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 7h - - 2024-09-21T02:56:46.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 7h - - 2024-09-21T02:56:46.021 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 0s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:56:46.021 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 19s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 0s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 19s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 0s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 0s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 19s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 0s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 0s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 19s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 0s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 0s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 0s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:56:46.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 0s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 19s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 19s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 19s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 19s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:56:46.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 19s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:56:46.333 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:56:46.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:56:46.333 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:56:46.334 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:56:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:56:46.601 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:56:46.601 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:56:46.601 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:56:46.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:56:46.914 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:56:46.914 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:46.914 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:46.914 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:46.914 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='client.82835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3582544074' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1076606840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='client.82835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3582544074' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:47.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1076606840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='client.82835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3582544074' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:56:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1076606840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:48 smithi184 ceph-mon[100622]: from='client.82859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:48 smithi184 ceph-mon[102844]: from='client.82859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:48 smithi196 ceph-mon[76349]: from='client.82859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:49 smithi184 ceph-mon[100622]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:49 smithi184 ceph-mon[102844]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:49 smithi196 ceph-mon[76349]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:51 smithi184 ceph-mon[100622]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:51 smithi184 ceph-mon[102844]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:51 smithi196 ceph-mon[76349]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:56:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:56:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:52.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:52.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:56:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:56:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:56:52.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:56:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:53 smithi184 ceph-mon[100622]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:53 smithi184 ceph-mon[102844]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:53 smithi196 ceph-mon[76349]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:55 smithi184 ceph-mon[100622]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:55 smithi184 ceph-mon[102844]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:55 smithi196 ceph-mon[76349]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:57.453 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:57 smithi184 ceph-mon[100622]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:57.453 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:57 smithi184 ceph-mon[102844]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:57 smithi196 ceph-mon[76349]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[102844]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[100622]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:56:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:59 smithi196 ceph-mon[76349]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:56:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:01 smithi184 ceph-mon[100622]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:01 smithi184 ceph-mon[102844]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:01.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:01 smithi196 ceph-mon[76349]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:57:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:02.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:02.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:02.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:03.508 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:03 smithi196 ceph-mon[76349]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:03.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:03 smithi184 ceph-mon[100622]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:03 smithi184 ceph-mon[102844]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:05.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:05 smithi196 ceph-mon[76349]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:05.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.471 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:05 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[100622]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[102844]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:05 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:06 smithi184 ceph-mon[100622]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:06 smithi184 ceph-mon[102844]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:06 smithi196 ceph-mon[76349]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:08 smithi196 ceph-mon[76349]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:08 smithi184 ceph-mon[100622]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:08 smithi184 ceph-mon[102844]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:11 smithi184 ceph-mon[100622]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:11 smithi184 ceph-mon[102844]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:11 smithi196 ceph-mon[76349]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:57:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:12.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:12.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:12.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:13.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:13 smithi184 ceph-mon[100622]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:13.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:13 smithi184 ceph-mon[102844]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:13 smithi196 ceph-mon[76349]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:15 smithi184 ceph-mon[100622]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:15 smithi184 ceph-mon[102844]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:15 smithi196 ceph-mon[76349]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:17.209 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:57:17.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:17 smithi184 ceph-mon[100622]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:17 smithi184 ceph-mon[102844]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:17 smithi196 ceph-mon[76349]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:17.738 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 7h - - 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 7h - - 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 12s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 10s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 12s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 10s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 12s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 12s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 10s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 12s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 12s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:57:17.739 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 10s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (6h) 12s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 12s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 12s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 12s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 10s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 10s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 10s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 10s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 10s ago 7h 79.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:57:17.740 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 10s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:57:18.055 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:57:18.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:18.057 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:57:18.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:57:18.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:57:18.057 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:57:18.331 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:57:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:57:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:57:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:57:18.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:57:18.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:18 smithi184 ceph-mon[102844]: from='client.72909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:18.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1449464378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:18.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:18 smithi184 ceph-mon[100622]: from='client.72909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:18.333 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1449464378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:18 smithi196 ceph-mon[76349]: from='client.72909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:18.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1449464378' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:18.646 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:57:18.646 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:18.647 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:18.647 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:18.647 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:19.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:19.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[100622]: from='client.82877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.398 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[100622]: from='client.82883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[100622]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3157270773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[102844]: from='client.82877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[102844]: from='client.82883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[102844]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:19.399 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3157270773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:19 smithi196 ceph-mon[76349]: from='client.82877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:19 smithi196 ceph-mon[76349]: from='client.82883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:19 smithi196 ceph-mon[76349]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3157270773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:20.485 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:20 smithi184 ceph-mon[100622]: from='client.82895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:20.486 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:20 smithi184 ceph-mon[102844]: from='client.82895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:20 smithi196 ceph-mon[76349]: from='client.82895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:21 smithi184 ceph-mon[100622]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:21 smithi184 ceph-mon[102844]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:21 smithi196 ceph-mon[76349]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:22] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T02:57:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:22.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:22.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:22.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:23 smithi184 ceph-mon[100622]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:23 smithi184 ceph-mon[102844]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:23 smithi196 ceph-mon[76349]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:25 smithi196 ceph-mon[76349]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:25 smithi184 ceph-mon[100622]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:25 smithi184 ceph-mon[102844]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[100622]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:26 smithi184 ceph-mon[102844]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:26 smithi196 ceph-mon[76349]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:29.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:28 smithi196 ceph-mon[76349]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:28 smithi184 ceph-mon[100622]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:28 smithi184 ceph-mon[102844]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:30.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:31.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:30 smithi196 ceph-mon[76349]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:30 smithi184 ceph-mon[100622]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:30 smithi184 ceph-mon[102844]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:57:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:32.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:32.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:32.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:33 smithi184 ceph-mon[102844]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:33 smithi184 ceph-mon[100622]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:33 smithi196 ceph-mon[76349]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:35 smithi184 ceph-mon[100622]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:35 smithi184 ceph-mon[102844]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:35 smithi196 ceph-mon[76349]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:37 smithi184 ceph-mon[100622]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:37 smithi184 ceph-mon[102844]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:37 smithi196 ceph-mon[76349]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:39 smithi184 ceph-mon[100622]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:39 smithi184 ceph-mon[102844]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:39 smithi196 ceph-mon[76349]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:41.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:41 smithi196 ceph-mon[76349]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:41 smithi184 ceph-mon[100622]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:41 smithi184 ceph-mon[102844]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:42.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T02:57:42.272 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:42.034Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:42.273 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:42.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:42.273 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:42.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:43 smithi184 ceph-mon[100622]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:43 smithi184 ceph-mon[102844]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:43 smithi196 ceph-mon[76349]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:44.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:45.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:45 smithi196 ceph-mon[76349]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:45 smithi184 ceph-mon[100622]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:45 smithi184 ceph-mon[102844]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:46.700 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.700 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.701 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.701 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:46 smithi196 ceph-mon[76349]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[100622]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:46 smithi184 ceph-mon[102844]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:57:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:57:48.942 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:57:49.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:49 smithi184 ceph-mon[100622]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:49 smithi184 ceph-mon[102844]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:49 smithi196 ceph-mon[76349]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:49.474 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:57:49.474 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 7h - - 2024-09-21T02:57:49.474 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 7h - - 2024-09-21T02:57:49.474 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 4s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 2s ago 7h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 4s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 2s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 4s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 4s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 4s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 4s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 2s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 4s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 4s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 4s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 2s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:57:49.475 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 2s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 2s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 2s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 2s ago 7h 73.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:57:49.476 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 2s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:57:49.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:57:49.784 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:57:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:57:50.055 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:57:50.055 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:57:50.055 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:57:50.363 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:57:50.363 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:50.363 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:50.364 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[102844]: from='client.72945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[102844]: from='client.72951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2379480591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[100622]: from='client.72945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[100622]: from='client.72951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.364 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2379480591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:50 smithi196 ceph-mon[76349]: from='client.72945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:50 smithi196 ceph-mon[76349]: from='client.72951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2379480591' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:51 smithi196 ceph-mon[76349]: from='client.82919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:51 smithi196 ceph-mon[76349]: from='client.82931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:51 smithi196 ceph-mon[76349]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:51.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1855892040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[100622]: from='client.82919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[100622]: from='client.82931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[100622]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1855892040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[102844]: from='client.82919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[102844]: from='client.82931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[102844]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1855892040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:57:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T02:57:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:52.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:52.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:57:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:57:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:57:52.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:57:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:53 smithi184 ceph-mon[100622]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:53 smithi184 ceph-mon[102844]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:53 smithi196 ceph-mon[76349]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:55 smithi184 ceph-mon[100622]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:55 smithi184 ceph-mon[102844]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:55 smithi196 ceph-mon[76349]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:57.547 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:57 smithi184 ceph-mon[100622]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:57 smithi184 ceph-mon[102844]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:57 smithi196 ceph-mon[76349]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[100622]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[102844]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:59.492 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:57:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:59 smithi196 ceph-mon[76349]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:57:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:01 smithi184 ceph-mon[100622]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:01 smithi184 ceph-mon[102844]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:01 smithi196 ceph-mon[76349]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:58:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:02.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:02.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:02.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:03 smithi184 ceph-mon[100622]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:03 smithi184 ceph-mon[102844]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:03 smithi196 ceph-mon[76349]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:05.371 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:05 smithi196 ceph-mon[76349]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:05 smithi184 ceph-mon[100622]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:05 smithi184 ceph-mon[102844]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:06 smithi196 ceph-mon[76349]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[100622]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:07.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:06 smithi184 ceph-mon[102844]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:09 smithi184 ceph-mon[100622]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:09 smithi184 ceph-mon[102844]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:09 smithi196 ceph-mon[76349]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:11 smithi184 ceph-mon[100622]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:11 smithi184 ceph-mon[102844]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:11 smithi196 ceph-mon[76349]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T02:58:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:12.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:12.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:12.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:13 smithi184 ceph-mon[100622]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:13 smithi184 ceph-mon[102844]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:13 smithi196 ceph-mon[76349]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:14.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 02:58:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T02:58:13.812107676Z level=info msg="Completed cleanup jobs" duration=11.93033ms 2024-09-21T02:58:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:15 smithi184 ceph-mon[100622]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:15 smithi184 ceph-mon[102844]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:15 smithi196 ceph-mon[76349]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:17.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:17 smithi184 ceph-mon[100622]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:17.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:17 smithi184 ceph-mon[102844]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:17 smithi196 ceph-mon[76349]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:19 smithi184 ceph-mon[100622]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:19 smithi184 ceph-mon[102844]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:19 smithi196 ceph-mon[76349]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:20.667 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 7h - - 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 15s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 15s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 14s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 15s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 15s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 14s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 15s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 15s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:58:21.241 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 14s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 15s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (6h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 15s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 15s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 14s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 14s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 14s ago 7h 73.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:58:21.242 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:58:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:21 smithi184 ceph-mon[100622]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:21 smithi184 ceph-mon[102844]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:58:21.592 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:58:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:21 smithi196 ceph-mon[76349]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:58:21.868 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:58:21.869 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:58:21.869 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:58:21.869 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:58:21.869 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:58:21.869 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:58:22.168 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:22.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:22.168 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:22.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:22.168 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:22.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:22.169 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:58:22.185 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:58:22.185 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:22.185 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:22.185 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:22.185 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[100622]: from='client.72981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[100622]: from='client.72987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[100622]: from='client.82955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/159927161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2757902531' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[102844]: from='client.72981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[102844]: from='client.72987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[102844]: from='client.82955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/159927161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:22.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2757902531' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:22 smithi196 ceph-mon[76349]: from='client.72981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:22 smithi196 ceph-mon[76349]: from='client.72987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:22 smithi196 ceph-mon[76349]: from='client.82955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/159927161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2757902531' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:23 smithi184 ceph-mon[100622]: from='client.82967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:23 smithi184 ceph-mon[100622]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:23 smithi184 ceph-mon[102844]: from='client.82967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:23 smithi184 ceph-mon[102844]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:23 smithi196 ceph-mon[76349]: from='client.82967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:23 smithi196 ceph-mon[76349]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:25 smithi184 ceph-mon[100622]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:25 smithi184 ceph-mon[102844]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:25 smithi196 ceph-mon[76349]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:26.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:26.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:26.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:26.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:26 smithi196 ceph-mon[76349]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[100622]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:27.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:26 smithi184 ceph-mon[102844]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:28 smithi196 ceph-mon[76349]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[100622]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:28 smithi184 ceph-mon[102844]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:31 smithi184 ceph-mon[100622]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:31 smithi184 ceph-mon[102844]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:31 smithi196 ceph-mon[76349]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:58:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:32.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:32.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:32.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:33 smithi184 ceph-mon[100622]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:33 smithi184 ceph-mon[102844]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:33 smithi196 ceph-mon[76349]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:35 smithi184 ceph-mon[100622]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:35 smithi184 ceph-mon[102844]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:35 smithi196 ceph-mon[76349]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:37 smithi184 ceph-mon[100622]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:37 smithi184 ceph-mon[102844]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:37 smithi196 ceph-mon[76349]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:39 smithi184 ceph-mon[100622]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:39 smithi184 ceph-mon[102844]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:39 smithi196 ceph-mon[76349]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:41.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:41 smithi184 ceph-mon[100622]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:41.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:41 smithi184 ceph-mon[102844]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:41 smithi196 ceph-mon[76349]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:58:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:42.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:42.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:42.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:43 smithi184 ceph-mon[100622]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:43 smithi184 ceph-mon[102844]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:43 smithi196 ceph-mon[76349]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:45 smithi184 ceph-mon[102844]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:45 smithi184 ceph-mon[100622]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:45 smithi196 ceph-mon[76349]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:46 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:46 smithi196 ceph-mon[76349]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[100622]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:47.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:47.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:46 smithi184 ceph-mon[102844]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.782 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:48.783 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:48 smithi196 ceph-mon[76349]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[100622]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:58:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:48 smithi184 ceph-mon[102844]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:51 smithi184 ceph-mon[100622]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:51 smithi184 ceph-mon[102844]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:51 smithi196 ceph-mon[76349]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:58:52] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T02:58:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:52.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:52.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:58:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:58:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:58:52.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:58:52.486 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 7h - - 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 7s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 5s ago 7h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 7s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 5s ago 7h 459M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 7s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 7s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 5s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:58:53.021 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 7s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 7s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 5s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 7s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (6h) 7s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 7s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 5s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 5s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 5s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 5s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 5s ago 7h 75.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:58:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:58:53.023 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 5s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:58:53.297 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:53 smithi184 ceph-mon[100622]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:53.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:53 smithi184 ceph-mon[102844]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:53.335 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:58:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:58:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:58:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:58:53.336 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:58:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:58:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:58:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:58:53.605 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:58:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:53 smithi196 ceph-mon[76349]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:53.914 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:58:53.914 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:53.914 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:53.914 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:53.914 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[100622]: from='client.82979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[100622]: from='client.73023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[100622]: from='client.82991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2819526905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1162809605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[102844]: from='client.82979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[102844]: from='client.73023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[102844]: from='client.82991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2819526905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:54 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1162809605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:54 smithi196 ceph-mon[76349]: from='client.82979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:54 smithi196 ceph-mon[76349]: from='client.73023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:54 smithi196 ceph-mon[76349]: from='client.82991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2819526905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:54 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1162809605' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:55 smithi184 ceph-mon[100622]: from='client.73041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:55 smithi184 ceph-mon[100622]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:55 smithi184 ceph-mon[102844]: from='client.73041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:55 smithi184 ceph-mon[102844]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:55 smithi196 ceph-mon[76349]: from='client.73041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:55 smithi196 ceph-mon[76349]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:57.491 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:57 smithi184 ceph-mon[100622]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:57.491 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:57 smithi184 ceph-mon[102844]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:57 smithi196 ceph-mon[76349]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[100622]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[102844]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:58:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:59 smithi196 ceph-mon[76349]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:58:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:01 smithi184 ceph-mon[100622]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:01.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:01 smithi184 ceph-mon[102844]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:01 smithi196 ceph-mon[76349]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:59:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:02.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:02.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:02.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:03.253 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:03 smithi184 ceph-mon[100622]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:03.254 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:03 smithi184 ceph-mon[102844]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:03 smithi196 ceph-mon[76349]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:05 smithi184 ceph-mon[100622]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:05 smithi184 ceph-mon[102844]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:05 smithi196 ceph-mon[76349]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:07.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:06 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:06 smithi196 ceph-mon[76349]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[100622]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:07.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:06 smithi184 ceph-mon[102844]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:08 smithi196 ceph-mon[76349]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[100622]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:09.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:09.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:08 smithi184 ceph-mon[102844]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:11 smithi184 ceph-mon[100622]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:11 smithi184 ceph-mon[102844]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:11 smithi196 ceph-mon[76349]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T02:59:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:12.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:12.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:12.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:13 smithi184 ceph-mon[100622]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:13 smithi184 ceph-mon[102844]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:13 smithi196 ceph-mon[76349]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:15 smithi184 ceph-mon[100622]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:15 smithi184 ceph-mon[102844]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:15 smithi196 ceph-mon[76349]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:17.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:17 smithi184 ceph-mon[100622]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:17.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:17 smithi184 ceph-mon[102844]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:17 smithi196 ceph-mon[76349]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:19 smithi184 ceph-mon[100622]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:19 smithi184 ceph-mon[102844]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:19 smithi196 ceph-mon[76349]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:21 smithi184 ceph-mon[100622]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:21 smithi184 ceph-mon[102844]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:21 smithi196 ceph-mon[76349]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:22.106 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T02:59:22.107 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:22.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:22.107 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:22.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:22.107 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:22.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:23.472 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:23 smithi184 ceph-mon[100622]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:23.472 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:23 smithi184 ceph-mon[102844]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:23 smithi196 ceph-mon[76349]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:24.206 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:59:24.737 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:59:24.737 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 7h - - 2024-09-21T02:59:24.737 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 7h - - 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 18s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 17s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 18s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 17s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 18s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 18s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 17s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 18s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 18s ago 7h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 17s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 18s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 18s ago 7h 85.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:59:24.738 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (6h) 18s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 17s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 17s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 17s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 17s ago 7h 75.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:59:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:59:25.081 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:59:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:59:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:59:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:59:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:59:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:59:25.083 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[102844]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[102844]: from='client.73053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2298186983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[100622]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[100622]: from='client.73053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:25.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2298186983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:59:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:59:25.390 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:59:25.390 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:59:25.390 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:59:25.390 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:59:25.390 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:59:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:25 smithi196 ceph-mon[76349]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:25 smithi196 ceph-mon[76349]: from='client.73053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2298186983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:25.710 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:59:25.710 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:25.710 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:25.710 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:25.711 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='client.83021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='client.73065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/282437994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='client.83021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='client.73065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/282437994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:26 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='client.83021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='client.73065 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/282437994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:26 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:27.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:27 smithi184 ceph-mon[100622]: from='client.73077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:27.516 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:27 smithi184 ceph-mon[100622]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:27.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:27 smithi184 ceph-mon[102844]: from='client.73077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:27.516 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:27 smithi184 ceph-mon[102844]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:27.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:27 smithi196 ceph-mon[76349]: from='client.73077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:27.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:27 smithi196 ceph-mon[76349]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:28 smithi196 ceph-mon[76349]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[100622]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:28 smithi184 ceph-mon[102844]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:30.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:30 smithi196 ceph-mon[76349]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:31.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:30 smithi184 ceph-mon[100622]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:31.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:30 smithi184 ceph-mon[102844]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:59:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:32.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:32.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:32.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:33 smithi184 ceph-mon[102844]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:33 smithi184 ceph-mon[100622]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:33 smithi196 ceph-mon[76349]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:35 smithi184 ceph-mon[100622]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:35 smithi184 ceph-mon[102844]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:35 smithi196 ceph-mon[76349]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:37.517 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:37 smithi184 ceph-mon[100622]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:37.518 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:37 smithi184 ceph-mon[102844]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:37 smithi196 ceph-mon[76349]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:39 smithi184 ceph-mon[102844]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:39 smithi184 ceph-mon[100622]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:39 smithi196 ceph-mon[76349]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:41 smithi184 ceph-mon[100622]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:41 smithi184 ceph-mon[102844]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:41 smithi196 ceph-mon[76349]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T02:59:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:42.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:42.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:42.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:43.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:43 smithi184 ceph-mon[100622]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:43.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:43 smithi184 ceph-mon[102844]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:43 smithi196 ceph-mon[76349]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:45 smithi184 ceph-mon[100622]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:45 smithi184 ceph-mon[102844]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:45 smithi196 ceph-mon[76349]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:47.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[100622]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[102844]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:47 smithi196 ceph-mon[76349]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:48 smithi196 ceph-mon[76349]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[100622]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T02:59:49.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:48 smithi184 ceph-mon[102844]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:51 smithi184 ceph-mon[100622]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:51 smithi184 ceph-mon[102844]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:51 smithi196 ceph-mon[76349]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 02:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:02:59:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T02:59:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:52.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:52.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T02:59:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 02:59:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T02:59:52.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T02:59:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:53 smithi184 ceph-mon[100622]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:53 smithi184 ceph-mon[102844]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:53 smithi196 ceph-mon[76349]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:55 smithi184 ceph-mon[100622]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:55 smithi184 ceph-mon[102844]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:55 smithi196 ceph-mon[76349]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:56.009 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T02:59:56.534 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 7h - - 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 10s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 8s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 10s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 8s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 10s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 8s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 10s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 10s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 8s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 10s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T02:59:56.535 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 10s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 10s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 8s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 8s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 8s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 8s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 8s ago 7h 77.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T02:59:56.536 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 8s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T02:59:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T02:59:56.844 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T02:59:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T02:59:57.113 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T02:59:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:59:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:57.426 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[100622]: from='client.83051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[100622]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[100622]: from='client.83057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2500641639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[102844]: from='client.83051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[102844]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[102844]: from='client.83057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.427 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2500641639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:57 smithi196 ceph-mon[76349]: from='client.83051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:57 smithi196 ceph-mon[76349]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:57 smithi196 ceph-mon[76349]: from='client.83057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2500641639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[100622]: from='client.83063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[100622]: from='client.73113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3187966203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[102844]: from='client.83063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[102844]: from='client.73113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3187966203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:58 smithi196 ceph-mon[76349]: from='client.83063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:58 smithi196 ceph-mon[76349]: from='client.73113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3187966203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:59 smithi196 ceph-mon[76349]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 02:59:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[100622]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[102844]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 02:59:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:00.264 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T02:59:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:01.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:01 smithi184 ceph-mon[100622]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:01 smithi184 ceph-mon[102844]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:01.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:01 smithi196 ceph-mon[76349]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:00:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:02.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:02.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:02.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:02.389 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 03:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T03:00:02.040Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726876800138 maxt=1726884000000 ulid=01J898B9TM7HNAQ3GBF2JGW9TE duration=35.846568ms 2024-09-21T03:00:02.390 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Sep 21 03:00:02 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-prometheus-a[65111]: ts=2024-09-21T03:00:02.043Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.186846ms 2024-09-21T03:00:03.424 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:03 smithi184 ceph-mon[100622]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:03.425 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:03 smithi184 ceph-mon[102844]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:03 smithi196 ceph-mon[76349]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:05.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:05 smithi184 ceph-mon[100622]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:05.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:05 smithi184 ceph-mon[102844]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:05 smithi196 ceph-mon[76349]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[100622]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[102844]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:07 smithi196 ceph-mon[76349]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:09.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:08 smithi196 ceph-mon[76349]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[100622]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:08 smithi184 ceph-mon[102844]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:11.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:11 smithi196 ceph-mon[76349]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:11 smithi184 ceph-mon[100622]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:11 smithi184 ceph-mon[102844]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:00:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:12.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:12.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:12.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:13 smithi184 ceph-mon[100622]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:13 smithi184 ceph-mon[102844]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:13 smithi196 ceph-mon[76349]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:15 smithi184 ceph-mon[100622]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:15 smithi184 ceph-mon[102844]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:15 smithi196 ceph-mon[76349]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:17 smithi184 ceph-mon[100622]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:17 smithi184 ceph-mon[102844]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:17 smithi196 ceph-mon[76349]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:19 smithi184 ceph-mon[102844]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:19 smithi184 ceph-mon[100622]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:19 smithi196 ceph-mon[76349]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:21 smithi184 ceph-mon[100622]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:21 smithi184 ceph-mon[102844]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:21 smithi196 ceph-mon[76349]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:00:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:22.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:22.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:22.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:23 smithi184 ceph-mon[100622]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:23 smithi184 ceph-mon[102844]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:23 smithi196 ceph-mon[76349]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:25 smithi184 ceph-mon[100622]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:25 smithi184 ceph-mon[102844]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:25 smithi196 ceph-mon[76349]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[100622]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[102844]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:27 smithi196 ceph-mon[76349]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:27.726 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 7h - - 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 7h - - 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 1s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 0s ago 7h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 1s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 0s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 1s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:00:28.257 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 1s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 0s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 1s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 1s ago 7h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 0s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 1s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 1s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 1s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 1s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 0s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 0s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 0s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 0s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 0s ago 7h 109M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:00:28.258 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:00:28.259 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:00:28.259 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:00:28.566 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:00:28.567 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:00:28.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:00:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:00:28.836 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:00:29.149 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:29.149 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:29.149 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:29.149 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:29.149 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='client.83087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2148069517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='client.83087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2148069517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='client.83087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2148069517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:30 smithi196 ceph-mon[76349]: from='client.83111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/38319369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[100622]: from='client.83111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/38319369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[102844]: from='client.83111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/38319369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:31 smithi184 ceph-mon[100622]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:31 smithi184 ceph-mon[102844]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:31 smithi196 ceph-mon[76349]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:00:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:32.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:32.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:32.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:33.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:33 smithi184 ceph-mon[100622]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:33.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:33 smithi184 ceph-mon[102844]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:33 smithi196 ceph-mon[76349]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:35 smithi184 ceph-mon[100622]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:35 smithi184 ceph-mon[102844]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:35 smithi196 ceph-mon[76349]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:37 smithi184 ceph-mon[100622]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:37 smithi184 ceph-mon[102844]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:37 smithi196 ceph-mon[76349]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:39 smithi184 ceph-mon[100622]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:39 smithi184 ceph-mon[102844]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:39 smithi196 ceph-mon[76349]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:41 smithi184 ceph-mon[100622]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:41 smithi184 ceph-mon[102844]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:41 smithi196 ceph-mon[76349]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:00:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:42.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:42.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:42.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:43.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:43 smithi184 ceph-mon[100622]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:43.547 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:43 smithi184 ceph-mon[102844]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:43 smithi196 ceph-mon[76349]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:45 smithi184 ceph-mon[100622]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:45 smithi184 ceph-mon[102844]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:45 smithi196 ceph-mon[76349]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[100622]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[102844]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:47 smithi196 ceph-mon[76349]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:47.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[100622]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[102844]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:49 smithi196 ceph-mon[76349]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:49.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:00:51.477 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:51 smithi196 ceph-mon[76349]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:51 smithi184 ceph-mon[100622]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:51 smithi184 ceph-mon[102844]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:00:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:00:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:52.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:52.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:00:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:00:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:00:52.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:00:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:53 smithi184 ceph-mon[100622]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:53 smithi184 ceph-mon[102844]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:53 smithi196 ceph-mon[76349]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:00:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:55 smithi184 ceph-mon[100622]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:55 smithi184 ceph-mon[102844]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:55.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:55 smithi196 ceph-mon[76349]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:57 smithi184 ceph-mon[100622]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:57 smithi184 ceph-mon[102844]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:57 smithi196 ceph-mon[76349]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:59.447 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:00:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[102844]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[100622]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:59.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:00:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:59 smithi196 ceph-mon[76349]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:00:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.972 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:00:59.972 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 7h - - 2024-09-21T03:00:59.972 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T03:00:59.972 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 13s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:00:59.972 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 11s ago 7h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 13s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 11s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 13s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 13s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 11s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 13s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 13s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 11s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 13s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 13s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 13s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 13s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (6h) 11s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:00:59.973 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 11s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 11s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 11s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 11s ago 7h 109M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:00:59.974 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 11s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:01:00.287 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:01:00.287 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:01:00.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:01:00.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:01:00.288 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:01:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:01:00.562 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:01:00.872 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:01:00.872 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:00.872 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: from='client.83123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: from='client.73167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: from='client.83135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1747875410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:01.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:01 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1259425666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: from='client.83123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: from='client.73167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: from='client.83135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1747875410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:01.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1259425666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: from='client.83123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: from='client.73167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: from='client.83135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1747875410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:01.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:01 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1259425666' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-mon[100622]: from='client.83147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-mon[102844]: from='client.83147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:01:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:02.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:02.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:02.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:02 smithi196 ceph-mon[76349]: from='client.83147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:03 smithi184 ceph-mon[100622]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:03 smithi184 ceph-mon[102844]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:03 smithi196 ceph-mon[76349]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:05 smithi184 ceph-mon[100622]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:05 smithi184 ceph-mon[102844]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:05 smithi196 ceph-mon[76349]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[100622]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[102844]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:07 smithi196 ceph-mon[76349]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[100622]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[102844]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:09 smithi196 ceph-mon[76349]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:11 smithi184 ceph-mon[100622]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:11 smithi184 ceph-mon[102844]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:11 smithi196 ceph-mon[76349]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:01:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:12.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:12.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:12.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:13 smithi184 ceph-mon[100622]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:13 smithi184 ceph-mon[102844]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:13.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:13 smithi196 ceph-mon[76349]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:15 smithi184 ceph-mon[100622]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:15 smithi184 ceph-mon[102844]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:15 smithi196 ceph-mon[76349]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:17 smithi184 ceph-mon[100622]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:17 smithi184 ceph-mon[102844]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:17 smithi196 ceph-mon[76349]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:19 smithi184 ceph-mon[100622]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:19 smithi184 ceph-mon[102844]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:19 smithi196 ceph-mon[76349]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:21 smithi184 ceph-mon[100622]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:21.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:21 smithi184 ceph-mon[102844]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:21 smithi196 ceph-mon[76349]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:01:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:22.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:22.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:22.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:23 smithi184 ceph-mon[100622]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:23 smithi184 ceph-mon[102844]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:23 smithi196 ceph-mon[76349]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:25 smithi184 ceph-mon[100622]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:25.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:25 smithi184 ceph-mon[102844]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:25.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:25 smithi196 ceph-mon[76349]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:27.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[100622]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:27.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[102844]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:27.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:27 smithi196 ceph-mon[76349]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:27.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:28.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:28.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:28 smithi184 ceph-mon[100622]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:28.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:28.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:28 smithi184 ceph-mon[102844]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:28 smithi196 ceph-mon[76349]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:29.785 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.786 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:29.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:30 smithi184 ceph-mon[100622]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:30 smithi184 ceph-mon[102844]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:30 smithi196 ceph-mon[76349]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:31.167 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:01:31.692 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:01:31.692 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 7h - - 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 7h - - 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 4s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 3s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 4s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 3s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 4s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 4s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 3s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 4s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 4s ago 7h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 3s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 4s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 4s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:01:31.693 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 4s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 3s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (6h) 3s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 3s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 3s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 3s ago 7h 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:01:31.694 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 3s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:01:31.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:31 smithi196 ceph-mon[76349]: from='client.83159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:31.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:31 smithi184 ceph-mon[100622]: from='client.83159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:31 smithi184 ceph-mon[102844]: from='client.83159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:01:32.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:01:32.005 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:01:32.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:01:32.273 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:01:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:01:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:01:32.274 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:01:32.274 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:32.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:32.274 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:32.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:32.274 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:32.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[100622]: from='client.83165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[100622]: from='client.83171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2629935659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[100622]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[100622]: from='client.73221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[102844]: from='client.83165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[102844]: from='client.83171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2629935659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[102844]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:32.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:32 smithi184 ceph-mon[102844]: from='client.73221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.589 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:01:32.590 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:32.590 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:32.590 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:32.590 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:32.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:32 smithi196 ceph-mon[76349]: from='client.83165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:32 smithi196 ceph-mon[76349]: from='client.83171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:32.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2629935659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:32.750 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:32 smithi196 ceph-mon[76349]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:32.751 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:32 smithi196 ceph-mon[76349]: from='client.73221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:33 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2605510769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:33 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2605510769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:33 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2605510769' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:34 smithi184 ceph-mon[100622]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:34 smithi184 ceph-mon[102844]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:34 smithi196 ceph-mon[76349]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:37 smithi184 ceph-mon[100622]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:37 smithi184 ceph-mon[102844]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:37 smithi196 ceph-mon[76349]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:39 smithi184 ceph-mon[100622]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:39 smithi184 ceph-mon[102844]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:39 smithi196 ceph-mon[76349]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:41.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:41 smithi184 ceph-mon[100622]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:41.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:41 smithi184 ceph-mon[102844]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:41.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:41 smithi196 ceph-mon[76349]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:42.297 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:01:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:42.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:42.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:42.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:43 smithi184 ceph-mon[100622]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:43 smithi184 ceph-mon[102844]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:43 smithi196 ceph-mon[76349]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:45 smithi196 ceph-mon[76349]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:45 smithi184 ceph-mon[100622]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:45.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:45 smithi184 ceph-mon[102844]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:46 smithi196 ceph-mon[76349]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:01:46.732 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:46 smithi184 ceph-mon[100622]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:01:46.733 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:46 smithi184 ceph-mon[102844]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:47 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:47 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:49.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[100622]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:49.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.212 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:49.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[102844]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:49.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:48 smithi196 ceph-mon[76349]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:01:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:51 smithi184 ceph-mon[100622]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:51 smithi184 ceph-mon[102844]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:51 smithi196 ceph-mon[76349]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:01:52] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T03:01:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:52.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:52.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:01:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:01:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:01:52.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:01:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:53 smithi196 ceph-mon[76349]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:53 smithi184 ceph-mon[100622]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:53 smithi184 ceph-mon[102844]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:55.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:55 smithi184 ceph-mon[100622]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:55.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:55 smithi184 ceph-mon[102844]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:55.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:55 smithi196 ceph-mon[76349]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:57 smithi184 ceph-mon[100622]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:57.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:57 smithi184 ceph-mon[102844]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:57 smithi196 ceph-mon[76349]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:58 smithi196 ceph-mon[76349]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:58 smithi184 ceph-mon[100622]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:58 smithi184 ceph-mon[102844]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:01:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:01:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:01:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:00 smithi196 ceph-mon[76349]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:00 smithi184 ceph-mon[100622]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:00 smithi184 ceph-mon[102844]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:02:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:02.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:02.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:02.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:02.884 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:02:03.412 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 16s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 94.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 16s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 14s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 16s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 16s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 14s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 16s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 16s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 14s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 16s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 16s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:02:03.413 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 16s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 16s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 14s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 14s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 14s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 14s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 14s ago 7h 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:02:03.414 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 14s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:02:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:03 smithi184 ceph-mon[100622]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:03.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:03 smithi184 ceph-mon[102844]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:03.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:03 smithi196 ceph-mon[76349]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:03.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:02:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:02:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:02:03.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:02:03.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:02:03.992 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:02:03.993 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:02:04.307 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:02:04.307 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:04.307 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[102844]: from='client.73233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[102844]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2574299268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[100622]: from='client.73233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[100622]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2574299268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:04 smithi196 ceph-mon[76349]: from='client.73233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:04 smithi196 ceph-mon[76349]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2574299268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[100622]: from='client.83207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[100622]: from='client.73257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[100622]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1861403024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[102844]: from='client.83207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[102844]: from='client.73257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[102844]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:05.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:05 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1861403024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:05 smithi196 ceph-mon[76349]: from='client.83207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:05 smithi196 ceph-mon[76349]: from='client.73257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:05 smithi196 ceph-mon[76349]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:05.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:05 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1861403024' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:07.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[100622]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:07.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.390 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[102844]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:07.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.391 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:07 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:07 smithi196 ceph-mon[76349]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:07 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:08.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:08.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:08 smithi184 ceph-mon[100622]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:08.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:08.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:08 smithi184 ceph-mon[102844]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:08 smithi196 ceph-mon[76349]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:10 smithi184 ceph-mon[100622]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:11.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:10 smithi184 ceph-mon[102844]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:11.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:10 smithi196 ceph-mon[76349]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:02:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:12.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:12.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:12.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:13 smithi184 ceph-mon[100622]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:13 smithi184 ceph-mon[102844]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:13.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:13 smithi196 ceph-mon[76349]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:15 smithi196 ceph-mon[76349]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:15 smithi184 ceph-mon[100622]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:15 smithi184 ceph-mon[102844]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:16 smithi196 ceph-mon[76349]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:16 smithi184 ceph-mon[100622]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:16 smithi184 ceph-mon[102844]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:19 smithi184 ceph-mon[102844]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:19 smithi184 ceph-mon[100622]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:19 smithi196 ceph-mon[76349]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:21 smithi196 ceph-mon[76349]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:21 smithi184 ceph-mon[100622]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:21 smithi184 ceph-mon[102844]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:02:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:22.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:22.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:22.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:23.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:23 smithi196 ceph-mon[76349]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:23 smithi184 ceph-mon[100622]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:23.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:23 smithi184 ceph-mon[102844]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:24 smithi196 ceph-mon[76349]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:24 smithi184 ceph-mon[100622]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:24 smithi184 ceph-mon[102844]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[100622]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[102844]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:27 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:27 smithi196 ceph-mon[76349]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:27.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:27 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.552 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.553 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:29.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:29.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:29.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:29.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:31 smithi184 ceph-mon[100622]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:31 smithi184 ceph-mon[102844]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:31 smithi196 ceph-mon[76349]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:02:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:32.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:32.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:32.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:33.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:33 smithi196 ceph-mon[76349]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:33.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:33 smithi184 ceph-mon[100622]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:33.742 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:33 smithi184 ceph-mon[102844]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:34.598 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:02:35.124 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 7h - - 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 7h - - 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 7s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 6s ago 7h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 7s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 6s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 7s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 7s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 6s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:02:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 7s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 7s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 6s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 7s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 7s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 7s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 6s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 6s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (6h) 6s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 6s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 6s ago 7h 65.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:02:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:02:35.418 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 6s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:02:35.445 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:02:35.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:02:35.445 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:02:35.446 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:02:35.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:35 smithi184 ceph-mon[102844]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:35.447 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:35 smithi184 ceph-mon[100622]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:35.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:35 smithi196 ceph-mon[76349]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:35.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:02:35.715 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:02:35.715 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:02:35.715 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:02:35.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:02:36.025 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:02:36.025 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:36.025 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:36 smithi196 ceph-mon[76349]: from='client.83231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:36 smithi196 ceph-mon[76349]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:36 smithi196 ceph-mon[76349]: from='client.83243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3801776446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2164531506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[100622]: from='client.83231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[100622]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[100622]: from='client.83243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3801776446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2164531506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[102844]: from='client.83231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[102844]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[102844]: from='client.83243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3801776446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2164531506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:37 smithi196 ceph-mon[76349]: from='client.73293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:37.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:37 smithi196 ceph-mon[76349]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:37.685 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:37 smithi184 ceph-mon[100622]: from='client.73293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:37.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:37 smithi184 ceph-mon[100622]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:37.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:37 smithi184 ceph-mon[102844]: from='client.73293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:37.686 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:37 smithi184 ceph-mon[102844]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:38 smithi196 ceph-mon[76349]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:38 smithi184 ceph-mon[100622]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:38 smithi184 ceph-mon[102844]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:41.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:41 smithi196 ceph-mon[76349]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:41 smithi184 ceph-mon[100622]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:41 smithi184 ceph-mon[102844]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:02:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:42.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:42.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:42.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:42 smithi196 ceph-mon[76349]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-mon[100622]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:42 smithi184 ceph-mon[102844]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:44.542 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:44.542 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:45 smithi196 ceph-mon[76349]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:45 smithi184 ceph-mon[100622]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:45 smithi184 ceph-mon[102844]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:47 smithi184 ceph-mon[100622]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:47 smithi184 ceph-mon[102844]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:47 smithi196 ceph-mon[76349]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:48 smithi196 ceph-mon[76349]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:48.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[100622]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:48.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:48 smithi184 ceph-mon[102844]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:50.093 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.093 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.094 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:50.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:02:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:51 smithi184 ceph-mon[102844]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:51 smithi184 ceph-mon[100622]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:51.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:51 smithi196 ceph-mon[76349]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:02:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:02:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:52.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:52.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:02:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:02:52.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:02:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:52 smithi196 ceph-mon[76349]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-mon[100622]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:52 smithi184 ceph-mon[102844]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:54 smithi196 ceph-mon[76349]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:54 smithi184 ceph-mon[100622]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:54 smithi184 ceph-mon[102844]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:56 smithi196 ceph-mon[76349]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:56 smithi184 ceph-mon[100622]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:56 smithi184 ceph-mon[102844]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:58 smithi196 ceph-mon[76349]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:58 smithi184 ceph-mon[100622]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:58 smithi184 ceph-mon[102844]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:02:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:02:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:02:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:00 smithi196 ceph-mon[76349]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:00 smithi184 ceph-mon[100622]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:00 smithi184 ceph-mon[102844]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:03:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:02.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:02.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:02.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:02 smithi196 ceph-mon[76349]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-mon[100622]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:02 smithi184 ceph-mon[102844]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:04 smithi196 ceph-mon[76349]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:04 smithi184 ceph-mon[100622]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:04 smithi184 ceph-mon[102844]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:06.322 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:03:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:06 smithi196 ceph-mon[76349]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:06 smithi184 ceph-mon[100622]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:06 smithi184 ceph-mon[102844]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.850 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:03:06.850 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T03:03:06.850 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 7h - - 2024-09-21T03:03:06.850 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:03:06.850 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 17s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 19s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 17s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 19s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 17s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 19s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 17s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 19s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 19s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 19s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 19s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:03:06.851 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 17s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 17s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 17s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 17s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 17s ago 7h 65.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:03:06.852 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 17s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:03:07.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:03:07.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:03:07.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:03:07.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:07.160 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:03:07.161 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:03:07.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[100622]: from='client.73305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[100622]: from='client.83273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[100622]: from='client.73317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3649491525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:07.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[102844]: from='client.73305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[102844]: from='client.83273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[102844]: from='client.73317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.436 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3649491525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:03:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:03:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:03:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:03:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:03:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:03:07.437 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:03:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:07 smithi196 ceph-mon[76349]: from='client.73305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:07 smithi196 ceph-mon[76349]: from='client.83273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:07 smithi196 ceph-mon[76349]: from='client.73317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3649491525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:07.749 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:03:07.749 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:07.749 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:07.749 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:07.749 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='client.83291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3018918029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[102844]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='client.83291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3018918029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:08.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:08 smithi184 ceph-mon[100622]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='client.83291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3018918029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:08 smithi196 ceph-mon[76349]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:10.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.305 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.305 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.306 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:10.394 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:11 smithi196 ceph-mon[76349]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:11.499 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:11 smithi184 ceph-mon[100622]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:11.499 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:11 smithi184 ceph-mon[102844]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:03:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:12.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:12.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:12.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:12 smithi196 ceph-mon[76349]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:12.688 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-mon[100622]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:12.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:12 smithi184 ceph-mon[102844]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:14.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:15.509 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:15 smithi196 ceph-mon[76349]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:15 smithi184 ceph-mon[100622]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:15 smithi184 ceph-mon[102844]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:16.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:16 smithi196 ceph-mon[76349]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:16 smithi184 ceph-mon[100622]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:16 smithi184 ceph-mon[102844]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:18 smithi196 ceph-mon[76349]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:18 smithi184 ceph-mon[102844]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:18 smithi184 ceph-mon[100622]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:20 smithi196 ceph-mon[76349]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:20 smithi184 ceph-mon[100622]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:20 smithi184 ceph-mon[102844]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:03:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:22.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:22.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:22.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:22 smithi196 ceph-mon[76349]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-mon[100622]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:22 smithi184 ceph-mon[102844]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:24 smithi196 ceph-mon[76349]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:24 smithi184 ceph-mon[100622]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:24 smithi184 ceph-mon[102844]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:26 smithi196 ceph-mon[76349]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:26 smithi184 ceph-mon[100622]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:26 smithi184 ceph-mon[102844]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:28 smithi196 ceph-mon[76349]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[100622]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:29.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:28 smithi184 ceph-mon[102844]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:29.942 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:30.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:30 smithi196 ceph-mon[76349]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:30.941 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:30 smithi184 ceph-mon[100622]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:30.941 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:30 smithi184 ceph-mon[102844]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:03:32.365 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:32.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:32.365 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:32.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:32.365 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:32.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:32 smithi196 ceph-mon[76349]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:32.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-mon[100622]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:32.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:32 smithi184 ceph-mon[102844]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:34 smithi196 ceph-mon[76349]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:34 smithi184 ceph-mon[100622]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:34 smithi184 ceph-mon[102844]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:36 smithi196 ceph-mon[76349]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:36 smithi184 ceph-mon[100622]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:36 smithi184 ceph-mon[102844]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.046 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:03:38.574 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:03:38.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 7h - - 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 10s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 9s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 10s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 9s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 10s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 9s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 10s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 10s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 9s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 10s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:03:38.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 10s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 10s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 9s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 9s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 9s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (6h) 9s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 9s ago 7h 66.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:03:38.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 9s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:03:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:38 smithi196 ceph-mon[76349]: from='client.73341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:38 smithi196 ceph-mon[76349]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:38 smithi196 ceph-mon[76349]: from='client.73347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[100622]: from='client.73341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[100622]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[100622]: from='client.73347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[102844]: from='client.73341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[102844]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:38 smithi184 ceph-mon[102844]: from='client.73347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:38.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:03:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:03:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:03:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:03:38.884 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:03:39.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:03:39.151 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:03:39.463 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:03:39.463 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:39.463 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:39.463 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:39.463 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:39.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[102844]: from='client.73353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:39.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1903493943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:39.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[102844]: from='client.83327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:39.464 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[100622]: from='client.73353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:39.464 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1903493943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:39.464 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:39 smithi184 ceph-mon[100622]: from='client.83327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:39 smithi196 ceph-mon[76349]: from='client.73353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1903493943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:39 smithi196 ceph-mon[76349]: from='client.83327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:40 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3888242394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:40 smithi196 ceph-mon[76349]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:40 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3888242394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:40 smithi184 ceph-mon[100622]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:40 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3888242394' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:40 smithi184 ceph-mon[102844]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:03:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:42.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:42.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:42.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:42 smithi196 ceph-mon[76349]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-mon[100622]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:42 smithi184 ceph-mon[102844]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:45 smithi184 ceph-mon[100622]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:45 smithi184 ceph-mon[102844]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:45 smithi196 ceph-mon[76349]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:46 smithi196 ceph-mon[76349]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:46 smithi184 ceph-mon[100622]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:46 smithi184 ceph-mon[102844]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[100622]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:49.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:48 smithi184 ceph-mon[102844]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:48 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:49.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:48 smithi196 ceph-mon[76349]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[100622]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:50 smithi184 ceph-mon[102844]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:03:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:50 smithi196 ceph-mon[76349]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:03:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:03:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:52.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:52.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:03:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:03:52.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:03:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:52 smithi196 ceph-mon[76349]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:52.778 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-mon[100622]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:52.778 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:52 smithi184 ceph-mon[102844]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:54 smithi196 ceph-mon[76349]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:54 smithi184 ceph-mon[100622]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:54 smithi184 ceph-mon[102844]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:56.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:56 smithi196 ceph-mon[76349]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:56 smithi184 ceph-mon[100622]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:56 smithi184 ceph-mon[102844]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:58.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:58 smithi196 ceph-mon[76349]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:58 smithi184 ceph-mon[100622]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:58 smithi184 ceph-mon[102844]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:03:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:03:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:03:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:00 smithi196 ceph-mon[76349]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:00 smithi184 ceph-mon[100622]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:00 smithi184 ceph-mon[102844]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:04:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:02.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:02.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:02.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:02 smithi196 ceph-mon[76349]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-mon[100622]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:02 smithi184 ceph-mon[102844]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:04 smithi196 ceph-mon[76349]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T03:04:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:04 smithi184 ceph-mon[100622]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T03:04:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:04 smithi184 ceph-mon[102844]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-21T03:04:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:06 smithi196 ceph-mon[76349]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T03:04:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:06 smithi184 ceph-mon[100622]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T03:04:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:06 smithi184 ceph-mon[102844]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T03:04:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:08 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:08 smithi196 ceph-mon[76349]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[100622]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:09.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:08 smithi184 ceph-mon[102844]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:04:09.761 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 7h - - 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 7h - - 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 2s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 0s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 2s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 0s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 2s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:04:10.286 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 2s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 2s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 2s ago 7h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 0s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 2s ago 7h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 2s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 2s ago 7h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 2s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 0s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:04:10.287 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 0s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 0s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 0s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 0s ago 7h 66.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:04:10.288 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (6h) 0s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='client.73377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: from='client.83351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[102844]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='client.73377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: from='client.83351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:10 smithi184 ceph-mon[100622]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:04:10.602 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:04:10.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:04:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:04:10.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:04:10.604 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:04:10.879 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:04:10.880 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:04:10.880 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:04:10.880 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:04:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='client.73377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: from='client.83351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:10 smithi196 ceph-mon[76349]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:04:11.203 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:04:11.203 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:11.203 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:11.203 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:11.203 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1640964660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[100622]: from='client.83363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1608940206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1640964660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[102844]: from='client.83363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:11 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1608940206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1640964660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:11 smithi196 ceph-mon[76349]: from='client.83363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:11 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1608940206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:12.275 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:04:12.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:12.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:12.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:12.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:12.275 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:12.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-mon[100622]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T03:04:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:12 smithi184 ceph-mon[102844]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T03:04:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:12 smithi196 ceph-mon[76349]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T03:04:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:15 smithi184 ceph-mon[100622]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T03:04:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:15 smithi184 ceph-mon[102844]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T03:04:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:15 smithi196 ceph-mon[76349]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T03:04:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:16 smithi196 ceph-mon[76349]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T03:04:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:16 smithi184 ceph-mon[100622]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T03:04:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:16 smithi184 ceph-mon[102844]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-21T03:04:18.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:18.547 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:18 smithi196 ceph-mon[76349]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:04:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:18 smithi184 ceph-mon[100622]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:04:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:18 smithi184 ceph-mon[102844]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:04:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:20 smithi196 ceph-mon[76349]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T03:04:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:20 smithi184 ceph-mon[100622]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T03:04:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:20 smithi184 ceph-mon[102844]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T03:04:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:04:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:22.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:22.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:22.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:22 smithi196 ceph-mon[76349]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:04:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-mon[100622]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:04:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:22 smithi184 ceph-mon[102844]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:04:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:24 smithi196 ceph-mon[76349]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:04:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:24 smithi184 ceph-mon[100622]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:04:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:24 smithi184 ceph-mon[102844]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:04:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:26 smithi196 ceph-mon[76349]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T03:04:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:26 smithi184 ceph-mon[100622]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T03:04:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:26 smithi184 ceph-mon[102844]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[100622]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:04:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:28 smithi184 ceph-mon[102844]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:04:29.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:29.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:29.342 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:28 smithi196 ceph-mon[76349]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:31 smithi184 ceph-mon[100622]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T03:04:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:31 smithi184 ceph-mon[102844]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T03:04:31.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:31 smithi196 ceph-mon[76349]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T03:04:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:04:32.372 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:32.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:32.372 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:32.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:32.372 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:32.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:32 smithi196 ceph-mon[76349]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:04:32.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-mon[100622]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:04:32.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:32 smithi184 ceph-mon[102844]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:04:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:34 smithi196 ceph-mon[76349]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T03:04:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:34 smithi184 ceph-mon[100622]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T03:04:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:34 smithi184 ceph-mon[102844]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-21T03:04:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:36 smithi196 ceph-mon[76349]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:04:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:36 smithi184 ceph-mon[100622]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:04:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:36 smithi184 ceph-mon[102844]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:04:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:38 smithi196 ceph-mon[76349]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:04:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:38 smithi184 ceph-mon[100622]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:04:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:38 smithi184 ceph-mon[102844]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 403 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:04:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:40 smithi196 ceph-mon[76349]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T03:04:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:40 smithi184 ceph-mon[100622]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T03:04:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:40 smithi184 ceph-mon[102844]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T03:04:41.500 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 7h - - 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 7h - - 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 14s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 12s ago 7h 96.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:04:42.026 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 14s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 12s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 14s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 14s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 12s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 14s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 14s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 12s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 14s ago 7h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 14s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 14s ago 7h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 14s ago 7h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 12s ago 7h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 12s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 12s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 12s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:04:42.027 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 12s ago 7h 66.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:04:42.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:04:42.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:04:42.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 14s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:04:42.028 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 12s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:04:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:04:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:42.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:42.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:42.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:04:42.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:04:42.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:04:42.609 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:04:42.610 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:04:42.610 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:04:42.610 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:04:42.610 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:04:42.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:04:42.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[102844]: from='client.83375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[102844]: from='client.83381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[102844]: from='client.73425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[102844]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:04:42.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3064820528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:42.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[100622]: from='client.83375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[100622]: from='client.83381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[100622]: from='client.73425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[100622]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:04:42.611 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3064820528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:42 smithi196 ceph-mon[76349]: from='client.83375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:42 smithi196 ceph-mon[76349]: from='client.83381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:42 smithi196 ceph-mon[76349]: from='client.73425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:42 smithi196 ceph-mon[76349]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 407 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:04:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3064820528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:42.921 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:04:42.921 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:42.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:42.921 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:42.921 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:43 smithi196 ceph-mon[76349]: from='client.73437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:43.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1829802044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:43 smithi184 ceph-mon[100622]: from='client.73437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1829802044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:43 smithi184 ceph-mon[102844]: from='client.73437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1829802044' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:44 smithi196 ceph-mon[76349]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:04:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:44 smithi184 ceph-mon[100622]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:04:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:44 smithi184 ceph-mon[102844]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:04:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:46 smithi196 ceph-mon[76349]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T03:04:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:46 smithi184 ceph-mon[100622]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T03:04:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:46 smithi184 ceph-mon[102844]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T03:04:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:49 smithi196 ceph-mon[76349]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T03:04:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[100622]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T03:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:49.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:49 smithi184 ceph-mon[102844]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[100622]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:50 smithi184 ceph-mon[102844]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:04:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:04:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:50 smithi196 ceph-mon[76349]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:04:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:04:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:04:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:52.065Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:52.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:04:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:04:52.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:04:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:52 smithi196 ceph-mon[76349]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:04:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-mon[100622]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:04:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:52 smithi184 ceph-mon[102844]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:04:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:54 smithi196 ceph-mon[76349]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T03:04:54.756 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:54 smithi184 ceph-mon[100622]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T03:04:54.757 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:54 smithi184 ceph-mon[102844]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T03:04:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:56 smithi196 ceph-mon[76349]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:56.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:56 smithi184 ceph-mon[100622]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:56 smithi184 ceph-mon[102844]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:58 smithi196 ceph-mon[76349]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:58 smithi184 ceph-mon[102844]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:58 smithi184 ceph-mon[100622]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:04:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:04:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:04:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:00 smithi196 ceph-mon[76349]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:00 smithi184 ceph-mon[102844]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:00 smithi184 ceph-mon[100622]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:05:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:02.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:02.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:02.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:02 smithi196 ceph-mon[76349]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-mon[100622]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:02 smithi184 ceph-mon[102844]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:04 smithi196 ceph-mon[76349]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:04.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:04 smithi184 ceph-mon[100622]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:04.797 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:04 smithi184 ceph-mon[102844]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:06 smithi196 ceph-mon[76349]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:06 smithi184 ceph-mon[102844]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:06 smithi184 ceph-mon[100622]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[100622]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:09 smithi184 ceph-mon[102844]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:09.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:09 smithi196 ceph-mon[76349]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[100622]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:11.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:10 smithi184 ceph-mon[102844]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:10 smithi196 ceph-mon[76349]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:05:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:12.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:12.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:12.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:12 smithi196 ceph-mon[76349]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-mon[100622]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:12 smithi184 ceph-mon[102844]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:13.223 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:05:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:13 smithi184 ceph-mon[100622]: from='client.83411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:13 smithi184 ceph-mon[102844]: from='client.83411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:13 smithi196 ceph-mon[76349]: from='client.83411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:13.766 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 7h - - 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 7h - - 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 5s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 3s ago 7h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 5s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 3s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 5s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 5s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 3s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 5s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 5s ago 7h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 3s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 5s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:05:13.767 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 5s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 5s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 5s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 3s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 3s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 3s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 3s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 3s ago 7h 66.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:05:13.768 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 3s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:05:14.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:05:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:05:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:05:14.086 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:05:14.372 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:05:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:05:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:05:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:05:14.373 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:05:14.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[100622]: from='client.83417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[100622]: from='client.83423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1537535482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[100622]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[102844]: from='client.83417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[102844]: from='client.83423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1537535482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:14.626 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:14 smithi184 ceph-mon[102844]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:14 smithi196 ceph-mon[76349]: from='client.83417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:14 smithi196 ceph-mon[76349]: from='client.83423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1537535482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:14 smithi196 ceph-mon[76349]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.707 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:05:14.707 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:14.708 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:15.704 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:15 smithi184 ceph-mon[100622]: from='client.83435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:15.705 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1386939953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:15.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:15 smithi184 ceph-mon[102844]: from='client.83435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:15.705 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1386939953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:15 smithi196 ceph-mon[76349]: from='client.83435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1386939953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:16 smithi184 ceph-mon[100622]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:16 smithi184 ceph-mon[102844]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:16 smithi196 ceph-mon[76349]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.561 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:18.562 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:18 smithi196 ceph-mon[76349]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:18 smithi184 ceph-mon[102844]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:18 smithi184 ceph-mon[100622]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:20.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:20 smithi196 ceph-mon[76349]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:20 smithi184 ceph-mon[100622]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:20 smithi184 ceph-mon[102844]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:05:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:22.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:22.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:22.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:22 smithi196 ceph-mon[76349]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-mon[100622]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:22 smithi184 ceph-mon[102844]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:24 smithi196 ceph-mon[76349]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:24 smithi184 ceph-mon[100622]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:24 smithi184 ceph-mon[102844]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:26 smithi196 ceph-mon[76349]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:26 smithi184 ceph-mon[100622]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:26 smithi184 ceph-mon[102844]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:28 smithi196 ceph-mon[76349]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:28 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[100622]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[102844]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:28.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:28 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[100622]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:31 smithi184 ceph-mon[102844]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:31.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:31 smithi196 ceph-mon[76349]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:32] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T03:05:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:32.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:32.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:32.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:32 smithi196 ceph-mon[76349]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:32.669 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-mon[100622]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:32.669 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:32 smithi184 ceph-mon[102844]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:34 smithi196 ceph-mon[76349]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:34 smithi184 ceph-mon[100622]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:34 smithi184 ceph-mon[102844]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:36 smithi196 ceph-mon[76349]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:36.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:36 smithi184 ceph-mon[100622]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:36.655 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:36 smithi184 ceph-mon[102844]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:38 smithi196 ceph-mon[76349]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:38 smithi184 ceph-mon[100622]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:38 smithi184 ceph-mon[102844]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:40.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:40 smithi196 ceph-mon[76349]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:40 smithi184 ceph-mon[100622]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:40 smithi184 ceph-mon[102844]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:42] "GET /metrics HTTP/1.1" 200 38952 "" "Prometheus/2.43.0" 2024-09-21T03:05:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:42.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:42.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:42.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:42 smithi196 ceph-mon[76349]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-mon[100622]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:42 smithi184 ceph-mon[102844]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:45.007 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:05:45.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:45 smithi184 ceph-mon[100622]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:45.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:45 smithi184 ceph-mon[102844]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 7h - - 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 7h - - 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 17s ago 7h 28.6M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 15s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 17s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 15s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 17s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 17s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 15s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 17s ago 7h 389M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 17s ago 7h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 15s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:05:45.547 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 17s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 17s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 17s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 17s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 15s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 15s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 15s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 15s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 15s ago 7h 64.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 17s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:05:45.548 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 15s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:05:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:45 smithi196 ceph-mon[76349]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:45.858 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:05:45.859 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:05:45.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:05:45.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:05:45.860 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:05:46.126 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:05:46.126 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:05:46.126 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:05:46.127 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:05:46.439 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:05:46.439 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:46.439 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:46.439 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:46.439 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:46.439 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[100622]: from='client.73485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[100622]: from='client.83453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.440 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2706117828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:46.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[102844]: from='client.73485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[102844]: from='client.83453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2706117828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:46 smithi196 ceph-mon[76349]: from='client.73485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:46 smithi196 ceph-mon[76349]: from='client.83453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2706117828' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[100622]: from='client.83459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[100622]: from='client.83471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[100622]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/548356733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[102844]: from='client.83459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[102844]: from='client.83471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[102844]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:47.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/548356733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:47.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:47 smithi196 ceph-mon[76349]: from='client.83459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:47 smithi196 ceph-mon[76349]: from='client.83471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:47 smithi196 ceph-mon[76349]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:47.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/548356733' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:48 smithi196 ceph-mon[76349]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:48 smithi184 ceph-mon[100622]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:48 smithi184 ceph-mon[102844]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[100622]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:51 smithi184 ceph-mon[102844]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:05:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:51 smithi196 ceph-mon[76349]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:05:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:05:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:52.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:52.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:05:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:05:52.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:05:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:52 smithi196 ceph-mon[76349]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-mon[100622]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:52 smithi184 ceph-mon[102844]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:54 smithi196 ceph-mon[76349]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:54.680 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:54 smithi184 ceph-mon[100622]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:54.680 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:54 smithi184 ceph-mon[102844]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:56 smithi184 ceph-mon[100622]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:56 smithi184 ceph-mon[102844]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:56 smithi196 ceph-mon[76349]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:58 smithi196 ceph-mon[76349]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:58 smithi184 ceph-mon[102844]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:58 smithi184 ceph-mon[100622]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:05:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:05:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:05:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:00.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:00 smithi196 ceph-mon[76349]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:00 smithi184 ceph-mon[100622]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:00 smithi184 ceph-mon[102844]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:06:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:02.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:02.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:02.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:02.557 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-mon[100622]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:02.557 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:02 smithi184 ceph-mon[102844]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:02 smithi196 ceph-mon[76349]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:04 smithi196 ceph-mon[76349]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:04 smithi184 ceph-mon[100622]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:04 smithi184 ceph-mon[102844]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:06 smithi196 ceph-mon[76349]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:06 smithi184 ceph-mon[102844]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:06 smithi184 ceph-mon[100622]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:08 smithi196 ceph-mon[76349]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:08.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:08 smithi184 ceph-mon[100622]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:08.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:08 smithi184 ceph-mon[102844]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[100622]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:11 smithi184 ceph-mon[102844]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:11 smithi196 ceph-mon[76349]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:06:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:12.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:12.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:12.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:12 smithi196 ceph-mon[76349]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-mon[100622]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:12 smithi184 ceph-mon[102844]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:15.292 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:15 smithi184 ceph-mon[100622]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:15.292 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:15 smithi184 ceph-mon[102844]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:15.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:15 smithi196 ceph-mon[76349]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:16.583 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:16 smithi184 ceph-mon[100622]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:16.583 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:16 smithi184 ceph-mon[102844]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:16 smithi196 ceph-mon[76349]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:16.762 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 7h - - 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 7h - - 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 8s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 7s ago 7h 94.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 8s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 7s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 8s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:06:17.316 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 8s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 7s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 8s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 8s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 7s ago 7h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 8s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 8s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 8s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 8s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 7s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 7s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 7s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 7s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 7s ago 7h 66.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:06:17.317 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:06:17.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:06:17.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:06:17.318 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 7s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[100622]: from='client.83483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[100622]: from='client.73527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[100622]: from='client.73533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[102844]: from='client.83483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[102844]: from='client.73527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:17 smithi184 ceph-mon[102844]: from='client.73533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.634 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:06:17.634 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:06:17.635 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:06:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:17 smithi196 ceph-mon[76349]: from='client.83483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:17 smithi196 ceph-mon[76349]: from='client.73527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:17 smithi196 ceph-mon[76349]: from='client.73533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:06:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:06:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:06:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:06:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:06:17.910 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1382565882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[102844]: from='client.83507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3078940719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[102844]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1382565882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[100622]: from='client.83507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3078940719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:18 smithi184 ceph-mon[100622]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1382565882' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:18 smithi196 ceph-mon[76349]: from='client.83507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3078940719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:18 smithi196 ceph-mon[76349]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:20.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:20 smithi196 ceph-mon[76349]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:20 smithi184 ceph-mon[100622]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:20 smithi184 ceph-mon[102844]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:06:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:22.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:22.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:22.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:22.626 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:22 smithi196 ceph-mon[76349]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-mon[100622]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:22 smithi184 ceph-mon[102844]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:24 smithi196 ceph-mon[76349]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:24 smithi184 ceph-mon[100622]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:24 smithi184 ceph-mon[102844]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:26 smithi196 ceph-mon[76349]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:26 smithi184 ceph-mon[100622]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:26 smithi184 ceph-mon[102844]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:28.769 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:28 smithi184 ceph-mon[100622]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:28.769 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:28 smithi184 ceph-mon[102844]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:28 smithi196 ceph-mon[76349]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[100622]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[102844]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:31 smithi196 ceph-mon[76349]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:06:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:32.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:32.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:32.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:32.551 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-mon[100622]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:32.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:32 smithi184 ceph-mon[102844]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:32 smithi196 ceph-mon[76349]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:34 smithi184 ceph-mon[100622]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:34 smithi184 ceph-mon[102844]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:34.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:34 smithi196 ceph-mon[76349]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:36 smithi184 ceph-mon[100622]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:36 smithi184 ceph-mon[102844]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:36.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:36 smithi196 ceph-mon[76349]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:38 smithi184 ceph-mon[100622]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:38 smithi184 ceph-mon[102844]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:38 smithi196 ceph-mon[76349]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:40 smithi184 ceph-mon[100622]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:40 smithi184 ceph-mon[102844]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:40.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:40 smithi196 ceph-mon[76349]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:06:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:42.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:42.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:42.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:42.647 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:42 smithi196 ceph-mon[76349]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-mon[100622]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:42 smithi184 ceph-mon[102844]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:45.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:45 smithi196 ceph-mon[76349]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:45 smithi184 ceph-mon[100622]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:45 smithi184 ceph-mon[102844]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:46 smithi184 ceph-mon[100622]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:46 smithi184 ceph-mon[102844]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:46.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:46 smithi196 ceph-mon[76349]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.529 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:06:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:48 smithi184 ceph-mon[100622]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:48 smithi184 ceph-mon[102844]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:48 smithi196 ceph-mon[76349]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 7h - - 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 7h - - 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 0s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 18s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 0s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 18s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 0s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 0s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:06:49.065 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 0s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 0s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 18s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 0s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 0s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 0s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 0s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 18s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 18s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 18s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 18s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 18s ago 7h 67.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:06:49.066 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:06:49.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 0s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:06:49.067 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 18s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:06:49.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:06:49.373 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:06:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:06:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:06:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:06:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:06:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:06:49.643 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:06:49.953 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:06:49.953 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:49.953 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:49.953 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:49.953 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='client.83519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='client.83531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/444791117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='client.83519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:49.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='client.83531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:49.955 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/444791117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='client.83519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='client.83531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/444791117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: from='client.73581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1329255174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: from='client.73581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1329255174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: from='client.73581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:51.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1329255174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:51.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:06:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:06:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:06:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:52.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:52.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:06:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:06:52.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:06:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-mon[100622]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:52 smithi184 ceph-mon[102844]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:52.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:52 smithi196 ceph-mon[76349]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:54.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:54 smithi196 ceph-mon[76349]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:54.797 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:54 smithi184 ceph-mon[100622]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:54 smithi184 ceph-mon[102844]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:56.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:56 smithi184 ceph-mon[100622]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:56.690 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:56 smithi184 ceph-mon[102844]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:56 smithi196 ceph-mon[76349]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:58.686 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:58 smithi184 ceph-mon[100622]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:58.687 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:58 smithi184 ceph-mon[102844]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:58 smithi196 ceph-mon[76349]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:59.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:06:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:06:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:06:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:00 smithi184 ceph-mon[102844]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:00 smithi184 ceph-mon[100622]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:00.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:00 smithi196 ceph-mon[76349]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:07:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:02.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:02.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:02.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:03 smithi184 ceph-mon[100622]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:03 smithi184 ceph-mon[102844]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:03 smithi196 ceph-mon[76349]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:05 smithi184 ceph-mon[100622]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:05 smithi184 ceph-mon[102844]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:05 smithi196 ceph-mon[76349]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:07.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:07 smithi184 ceph-mon[100622]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:07 smithi184 ceph-mon[102844]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:07 smithi196 ceph-mon[76349]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[100622]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[102844]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:10 smithi196 ceph-mon[76349]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:11 smithi196 ceph-mon[76349]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:11.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[102844]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[100622]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:07:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:12.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:12.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:12.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:13 smithi184 ceph-mon[100622]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:13 smithi184 ceph-mon[102844]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:13 smithi196 ceph-mon[76349]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:15 smithi184 ceph-mon[100622]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:15 smithi184 ceph-mon[102844]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:16.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:15 smithi196 ceph-mon[76349]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:17 smithi184 ceph-mon[100622]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:17 smithi184 ceph-mon[102844]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:17 smithi196 ceph-mon[76349]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:19.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:19 smithi184 ceph-mon[100622]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:20.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:19 smithi184 ceph-mon[102844]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:19 smithi196 ceph-mon[76349]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:20.253 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 7h - - 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 7h - - 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 11s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 10s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 11s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 10s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:07:20.783 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 11s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 11s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 10s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 11s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 11s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 10s ago 7h 17.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 11s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 11s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 11s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 11s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 10s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 10s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 10s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 10s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:07:20.784 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 10s ago 7h 65.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:07:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:07:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:07:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:07:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 10s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:21.094 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:07:21.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:07:21.365 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:07:21.365 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:07:21.366 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:07:21.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:21 smithi184 ceph-mon[100622]: from='client.73593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:21.367 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2702475201' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:21.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:21 smithi184 ceph-mon[102844]: from='client.73593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:21.367 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2702475201' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:21 smithi196 ceph-mon[76349]: from='client.73593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:21.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2702475201' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:21.673 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:07:21.673 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:21.673 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:07:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[100622]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[100622]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[100622]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2591043009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[102844]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[102844]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[102844]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2591043009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:22.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:22.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:22.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:22.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:22 smithi196 ceph-mon[76349]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:22.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:22 smithi196 ceph-mon[76349]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:22 smithi196 ceph-mon[76349]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:22.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:22 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2591043009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:23.528 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:23 smithi196 ceph-mon[76349]: from='client.83579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:23 smithi184 ceph-mon[100622]: from='client.83579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:23 smithi184 ceph-mon[102844]: from='client.83579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:24.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:24 smithi196 ceph-mon[76349]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:24 smithi184 ceph-mon[100622]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:24 smithi184 ceph-mon[102844]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:26 smithi184 ceph-mon[100622]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:26 smithi184 ceph-mon[102844]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:26 smithi196 ceph-mon[76349]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:28 smithi184 ceph-mon[100622]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:28 smithi184 ceph-mon[102844]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:28 smithi196 ceph-mon[76349]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:30 smithi184 ceph-mon[100622]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:30 smithi184 ceph-mon[102844]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:30 smithi196 ceph-mon[76349]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:32.079 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[100622]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[102844]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.080 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:31 smithi196 ceph-mon[76349]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:32.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:32.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:32.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:32.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:32.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:32.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:33 smithi196 ceph-mon[76349]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:33 smithi184 ceph-mon[100622]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:33 smithi184 ceph-mon[102844]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:35 smithi196 ceph-mon[76349]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:35 smithi184 ceph-mon[100622]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:35 smithi184 ceph-mon[102844]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:37 smithi184 ceph-mon[100622]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:37 smithi184 ceph-mon[102844]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:37 smithi196 ceph-mon[76349]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:39.088 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:39.088 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:39 smithi196 ceph-mon[76349]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:40.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:39 smithi184 ceph-mon[100622]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:40.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:39 smithi184 ceph-mon[102844]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:41 smithi196 ceph-mon[76349]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:41 smithi184 ceph-mon[100622]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:41 smithi184 ceph-mon[102844]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:07:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:42.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:42.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:42.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:44.100 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:43 smithi196 ceph-mon[76349]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:43 smithi184 ceph-mon[100622]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:43 smithi184 ceph-mon[102844]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:45.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:45 smithi196 ceph-mon[76349]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:45 smithi184 ceph-mon[100622]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:45 smithi184 ceph-mon[102844]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:47 smithi196 ceph-mon[76349]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:47 smithi184 ceph-mon[100622]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:47 smithi184 ceph-mon[102844]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:49.280 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:49.280 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[100622]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[102844]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:50 smithi196 ceph-mon[76349]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:51.969 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[100622]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:07:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:07:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[102844]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:52.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:52.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:07:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:07:52.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:07:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:51 smithi196 ceph-mon[76349]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 7h - - 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 7h - - 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 3s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 1s ago 7h 93.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 3s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:07:52.497 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 1s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 3s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 3s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 1s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 3s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 3s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 1s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 3s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 3s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 3s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 3s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 1s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 1s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 1s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:07:52.498 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 1s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:07:52.499 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 1s ago 7h 67.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:07:52.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:07:52.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:07:52.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 3s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:07:52.499 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 1s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:07:52.813 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:07:52.813 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:07:52.814 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:07:52.815 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:07:52.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:07:52.815 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:07:53.086 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:07:53.087 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:07:53.087 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:07:53.087 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:07:53.087 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:07:53.087 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[100622]: from='client.73629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.087 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[100622]: from='client.83597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.087 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1135732648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:53.087 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[102844]: from='client.73629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.088 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[102844]: from='client.83597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.088 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1135732648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:53.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:52 smithi196 ceph-mon[76349]: from='client.73629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:52 smithi196 ceph-mon[76349]: from='client.83597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1135732648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:53.398 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:07:53.398 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:53.398 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:53.398 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:53.398 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[102844]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[102844]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[102844]: from='client.83615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3183611977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[100622]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[100622]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[100622]: from='client.83615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3183611977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:53 smithi196 ceph-mon[76349]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:53 smithi196 ceph-mon[76349]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:53 smithi196 ceph-mon[76349]: from='client.83615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3183611977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:55 smithi184 ceph-mon[100622]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:55 smithi184 ceph-mon[102844]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:55 smithi196 ceph-mon[76349]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:57 smithi184 ceph-mon[100622]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:57 smithi184 ceph-mon[102844]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:57 smithi196 ceph-mon[76349]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:59.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:59 smithi184 ceph-mon[100622]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:07:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:59 smithi184 ceph-mon[102844]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:00.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:07:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:59 smithi196 ceph-mon[76349]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:07:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:01 smithi184 ceph-mon[100622]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:08:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:01 smithi184 ceph-mon[102844]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:02.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:02.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:02.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:01 smithi196 ceph-mon[76349]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:03 smithi196 ceph-mon[76349]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:03 smithi184 ceph-mon[100622]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:03 smithi184 ceph-mon[102844]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:06.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:05 smithi196 ceph-mon[76349]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:05 smithi184 ceph-mon[100622]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:05 smithi184 ceph-mon[102844]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:07 smithi184 ceph-mon[100622]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:07 smithi184 ceph-mon[102844]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:07 smithi196 ceph-mon[76349]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[100622]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[102844]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:09 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:09 smithi196 ceph-mon[76349]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:09 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[100622]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[102844]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:12.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:12.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:12.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:12 smithi196 ceph-mon[76349]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:14.091 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 03:08:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T03:08:13.81285814Z level=info msg="Completed cleanup jobs" duration=12.916168ms 2024-09-21T03:08:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:14 smithi196 ceph-mon[76349]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:14 smithi184 ceph-mon[100622]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:14 smithi184 ceph-mon[102844]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:15.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:16 smithi196 ceph-mon[76349]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:16 smithi184 ceph-mon[100622]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:16 smithi184 ceph-mon[102844]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:18 smithi184 ceph-mon[100622]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:18 smithi184 ceph-mon[102844]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:18 smithi196 ceph-mon[76349]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:19.289 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:19.289 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:20 smithi184 ceph-mon[100622]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:20 smithi184 ceph-mon[102844]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:20 smithi196 ceph-mon[76349]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-mon[100622]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-mon[102844]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:08:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:22.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:22.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:22.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:22 smithi196 ceph-mon[76349]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:23.685 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 7h - - 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 7h - - 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 14s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 13s ago 7h 94.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 14s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 13s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 14s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:08:24.213 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 14s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 13s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 14s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 14s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 13s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 14s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 14s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 14s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 14s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 13s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 13s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 13s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 13s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 13s ago 7h 67.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:08:24.214 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:08:24.215 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:08:24.215 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 14s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:08:24.215 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 13s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:08:24.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:24 smithi196 ceph-mon[76349]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:24.525 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:08:24.525 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:08:24.525 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:08:24.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:08:24.526 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:08:24.527 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:24 smithi184 ceph-mon[102844]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:24.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:24 smithi184 ceph-mon[100622]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:08:24.795 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:08:24.796 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:08:24.796 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:08:25.105 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:08:25.105 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:25.105 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:25 smithi196 ceph-mon[76349]: from='client.73665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:25 smithi196 ceph-mon[76349]: from='client.73671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:25 smithi196 ceph-mon[76349]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4209587022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:25.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:25 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1082021794' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[100622]: from='client.73665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[100622]: from='client.73671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[100622]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4209587022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:25.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1082021794' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[102844]: from='client.73665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[102844]: from='client.73671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[102844]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4209587022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:25.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:25 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1082021794' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:26.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:26 smithi196 ceph-mon[76349]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:26.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:26 smithi196 ceph-mon[76349]: from='client.73689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:26 smithi184 ceph-mon[102844]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:26 smithi184 ceph-mon[102844]: from='client.73689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:26 smithi184 ceph-mon[100622]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:26 smithi184 ceph-mon[100622]: from='client.73689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:28 smithi184 ceph-mon[102844]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:28 smithi184 ceph-mon[100622]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:28 smithi196 ceph-mon[76349]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:29.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:29.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:29.539 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:30 smithi196 ceph-mon[76349]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[102844]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[100622]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[100622]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[102844]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:32.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:32.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:32.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:32 smithi196 ceph-mon[76349]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:34 smithi184 ceph-mon[100622]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:34 smithi184 ceph-mon[102844]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:34 smithi196 ceph-mon[76349]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:36 smithi184 ceph-mon[100622]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:36 smithi184 ceph-mon[102844]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:36 smithi196 ceph-mon[76349]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:38 smithi184 ceph-mon[100622]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:38.440 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:38 smithi184 ceph-mon[102844]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:38 smithi196 ceph-mon[76349]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:40 smithi184 ceph-mon[102844]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:40 smithi184 ceph-mon[100622]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:40 smithi196 ceph-mon[76349]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:42.219 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:08:42.220 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:42.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:42.220 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:42.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:42.220 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:42.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-mon[102844]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:42 smithi184 ceph-mon[100622]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:42 smithi196 ceph-mon[76349]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:44 smithi184 ceph-mon[102844]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:44 smithi184 ceph-mon[100622]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:44 smithi196 ceph-mon[76349]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:46.363 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:46 smithi196 ceph-mon[76349]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:46 smithi184 ceph-mon[102844]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:46 smithi184 ceph-mon[100622]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:48 smithi184 ceph-mon[102844]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:48 smithi184 ceph-mon[100622]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:48 smithi196 ceph-mon[76349]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:49.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:49.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[100622]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[102844]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:50 smithi196 ceph-mon[76349]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:08:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:08:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:52.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:52.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:08:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:08:52.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:08:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:52 smithi196 ceph-mon[76349]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[100622]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[102844]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:08:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:54 smithi196 ceph-mon[76349]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:54 smithi184 ceph-mon[100622]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:54 smithi184 ceph-mon[102844]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:55.408 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:08:55.935 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 7h - - 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 7h - - 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 6s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 4s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 6s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 4s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 6s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 6s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 4s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 6s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 6s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 4s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 6s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:08:55.936 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 6s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 6s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 4s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 4s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 4s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 4s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 4s ago 7h 69.6M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:08:55.937 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 4s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:08:56.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:08:56.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:08:56.521 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:08:56.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:08:56.522 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:56 smithi184 ceph-mon[100622]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:56.523 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/14687207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:56.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:56 smithi184 ceph-mon[102844]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:56.523 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/14687207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:56 smithi196 ceph-mon[76349]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/14687207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:56.832 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:08:56.832 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:56.832 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:56.832 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:56.832 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:57.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:57 smithi196 ceph-mon[76349]: from='client.73701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:57 smithi196 ceph-mon[76349]: from='client.83669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:57 smithi196 ceph-mon[76349]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:57 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1099685643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[100622]: from='client.73701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[100622]: from='client.83669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[100622]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1099685643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[102844]: from='client.73701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[102844]: from='client.83669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[102844]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:57 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1099685643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:58 smithi196 ceph-mon[76349]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:58 smithi196 ceph-mon[76349]: from='client.83687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:58 smithi184 ceph-mon[100622]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:58 smithi184 ceph-mon[100622]: from='client.83687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:58 smithi184 ceph-mon[102844]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:08:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:58 smithi184 ceph-mon[102844]: from='client.83687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:08:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:08:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:08:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:00.564 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:00 smithi184 ceph-mon[100622]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:00.564 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:00 smithi184 ceph-mon[102844]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:00 smithi196 ceph-mon[76349]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:02.212 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:09:02.212 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:02.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:02.212 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:02.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:02.212 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:02.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-mon[100622]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:02 smithi184 ceph-mon[102844]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:02 smithi196 ceph-mon[76349]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:04 smithi196 ceph-mon[76349]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:04 smithi184 ceph-mon[100622]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:04 smithi184 ceph-mon[102844]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:06 smithi196 ceph-mon[76349]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:06 smithi184 ceph-mon[100622]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:06 smithi184 ceph-mon[102844]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:08 smithi196 ceph-mon[76349]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:08 smithi184 ceph-mon[100622]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:08 smithi184 ceph-mon[102844]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:10 smithi196 ceph-mon[76349]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[102844]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[100622]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:11.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:11 smithi196 ceph-mon[76349]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:11 smithi184 ceph-mon[100622]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:11 smithi184 ceph-mon[102844]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:09:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:12.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:12.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:12.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:13 smithi184 ceph-mon[100622]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:13 smithi184 ceph-mon[102844]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:13 smithi196 ceph-mon[76349]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:16 smithi184 ceph-mon[100622]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:16 smithi184 ceph-mon[102844]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:16 smithi196 ceph-mon[76349]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:18 smithi184 ceph-mon[100622]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:18 smithi184 ceph-mon[102844]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:18 smithi196 ceph-mon[76349]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:19.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:19.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:20 smithi184 ceph-mon[100622]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:20 smithi184 ceph-mon[102844]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:20 smithi196 ceph-mon[76349]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:22.266 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:09:22.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:22.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:22.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:22.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:22.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:22.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-mon[102844]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:22 smithi184 ceph-mon[100622]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:22 smithi196 ceph-mon[76349]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:24 smithi184 ceph-mon[100622]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:24 smithi184 ceph-mon[102844]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:24 smithi196 ceph-mon[76349]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:26.502 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:26 smithi196 ceph-mon[76349]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:26 smithi184 ceph-mon[100622]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:26 smithi184 ceph-mon[102844]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:27.129 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 7h - - 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 7h - - 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 17s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 16s ago 7h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 17s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 16s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 17s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:09:27.653 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 17s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 16s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 17s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 17s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 16s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 17s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 17s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 17s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 17s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 16s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 16s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 16s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 16s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 16s ago 7h 69.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:09:27.654 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:09:27.655 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:09:27.655 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 17s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:09:27.655 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 16s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:09:27.961 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:09:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:09:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:09:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:09:27.963 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:09:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:09:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:09:28.235 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:09:28.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:28 smithi196 ceph-mon[76349]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:28 smithi196 ceph-mon[76349]: from='client.83699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:28.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:28 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3511882130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:28.545 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:09:28.546 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:28.546 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[102844]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[102844]: from='client.83699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3511882130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[100622]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[100622]: from='client.83699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:28.546 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:28 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3511882130' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='client.83711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='client.83723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/33976059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='client.83711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='client.83723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/33976059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:29.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='client.83711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='client.83723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/33976059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[100622]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[102844]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:30 smithi196 ceph-mon[76349]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.287 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:09:32.288 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:32.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:32.288 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:32.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:32.288 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:32.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[102844]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[100622]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:32 smithi196 ceph-mon[76349]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:34 smithi184 ceph-mon[100622]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:34 smithi184 ceph-mon[102844]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:34 smithi196 ceph-mon[76349]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:36 smithi184 ceph-mon[100622]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:36 smithi184 ceph-mon[102844]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:36 smithi196 ceph-mon[76349]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:38 smithi196 ceph-mon[76349]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:38 smithi184 ceph-mon[100622]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:38 smithi184 ceph-mon[102844]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:40 smithi196 ceph-mon[76349]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:40 smithi184 ceph-mon[100622]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:40 smithi184 ceph-mon[102844]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:09:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:42.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:42.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:42.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:42.590 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-mon[100622]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:42.591 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:42 smithi184 ceph-mon[102844]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:42 smithi196 ceph-mon[76349]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:44 smithi196 ceph-mon[76349]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:44 smithi184 ceph-mon[102844]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:44 smithi184 ceph-mon[100622]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:46 smithi196 ceph-mon[76349]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:46 smithi184 ceph-mon[100622]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:46 smithi184 ceph-mon[102844]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:48 smithi196 ceph-mon[76349]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:48 smithi184 ceph-mon[100622]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:48 smithi184 ceph-mon[102844]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:49.525 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:50 smithi196 ceph-mon[76349]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:50 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[102844]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[100622]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:50 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:09:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:09:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:52.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:52.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:09:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:09:52.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:09:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:52 smithi196 ceph-mon[76349]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[102844]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[100622]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:52.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:09:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:54 smithi196 ceph-mon[76349]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:54 smithi184 ceph-mon[102844]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:54 smithi184 ceph-mon[100622]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:56 smithi196 ceph-mon[76349]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:56 smithi184 ceph-mon[102844]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:56 smithi184 ceph-mon[100622]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:58 smithi196 ceph-mon[76349]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:09:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:58 smithi184 ceph-mon[100622]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:09:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:58 smithi184 ceph-mon[102844]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:09:58.842 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 7h - - 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 7h - - 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 9s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 7s ago 7h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 9s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 7s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 9s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:09:59.365 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 9s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 7s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 9s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 9s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 7s ago 7h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 9s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 9s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 9s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 9s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 7s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 7s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 7s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 7s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 7s ago 7h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:09:59.366 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:09:59.367 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:09:59.367 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 9s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:09:59.367 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 7s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:09:59.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:09:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:09:59.676 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:09:59.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:09:59.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:09:59.677 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:09:59.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:59.677 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:09:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:59.677 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:59.677 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:09:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:09:59.945 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:09:59.946 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:09:59.946 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:09:59.946 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:09:59.946 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:10:00.255 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.255 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:00.255 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:00.255 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.256 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T03:09:59.999+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: from='client.73773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: from='client.83741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: from='client.83747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/300689557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/965332151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: from='client.73773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: from='client.83741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: from='client.83747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/300689557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/965332151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: from='client.73773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: from='client.83741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: from='client.83747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/300689557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/965332151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:01.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:01 smithi196 ceph-mon[76349]: from='client.73797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:01 smithi184 ceph-mon[100622]: from='client.73797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:01 smithi184 ceph-mon[102844]: from='client.73797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:02.273 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:10:02.273 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:02.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:02.273 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:02.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:02.274 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:02.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-mon[102844]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:02 smithi184 ceph-mon[100622]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:02 smithi196 ceph-mon[76349]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:04 smithi196 ceph-mon[76349]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:04.717 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:04 smithi184 ceph-mon[100622]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:04.717 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:04 smithi184 ceph-mon[102844]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:06 smithi196 ceph-mon[76349]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:06 smithi184 ceph-mon[100622]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:06 smithi184 ceph-mon[102844]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:08 smithi196 ceph-mon[76349]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:08 smithi184 ceph-mon[100622]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:08 smithi184 ceph-mon[102844]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[102844]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[100622]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:10 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:10 smithi196 ceph-mon[76349]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:10 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:12.089Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:12.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:12.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:12 smithi196 ceph-mon[76349]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[100622]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[102844]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:13 smithi184 ceph-mon[100622]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:13 smithi184 ceph-mon[102844]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:13 smithi196 ceph-mon[76349]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:15 smithi184 ceph-mon[102844]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:15 smithi184 ceph-mon[100622]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:15 smithi196 ceph-mon[76349]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:17 smithi184 ceph-mon[100622]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:17 smithi184 ceph-mon[102844]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:17 smithi196 ceph-mon[76349]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:19 smithi184 ceph-mon[100622]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:19 smithi184 ceph-mon[102844]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:19 smithi196 ceph-mon[76349]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:21 smithi184 ceph-mon[100622]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:21.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:21 smithi184 ceph-mon[102844]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:21 smithi196 ceph-mon[76349]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:22.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:22.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:22.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:23.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:23 smithi184 ceph-mon[100622]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:23.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:23 smithi184 ceph-mon[102844]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:23 smithi196 ceph-mon[76349]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:25.770 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:25 smithi184 ceph-mon[100622]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:25.770 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:25 smithi184 ceph-mon[102844]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:25 smithi196 ceph-mon[76349]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:27.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:27 smithi184 ceph-mon[100622]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:27 smithi184 ceph-mon[102844]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:27 smithi196 ceph-mon[76349]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:28.743 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:29.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:29 smithi196 ceph-mon[76349]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:29.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:29 smithi184 ceph-mon[100622]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:29 smithi184 ceph-mon[102844]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:30.546 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 7h - - 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 7h - - 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 0s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 19s ago 7h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 0s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 19s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 0s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:10:31.071 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 0s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 19s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 0s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 0s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 19s ago 7h 18.0M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 0s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 0s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 0s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 0s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 19s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 19s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 19s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 19s ago 7h 70.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:10:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:10:31.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 0s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:10:31.073 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 19s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:10:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:10:31.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:10:31.382 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.383 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.383 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:31.650 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:10:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:10:31.651 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:10:31.958 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:31.958 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:31.958 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:10:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:32.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:32.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:32.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='client.73809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='client.83783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2965452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/301544103' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='client.73809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='client.83783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2965452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/301544103' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='client.73809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='client.83783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2965452541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/301544103' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:33 smithi196 ceph-mon[76349]: from='client.73833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:33 smithi184 ceph-mon[100622]: from='client.73833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:33 smithi184 ceph-mon[102844]: from='client.73833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:34 smithi196 ceph-mon[76349]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:34 smithi184 ceph-mon[100622]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:34 smithi184 ceph-mon[102844]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:36 smithi196 ceph-mon[76349]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:36 smithi184 ceph-mon[100622]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:36 smithi184 ceph-mon[102844]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:38 smithi184 ceph-mon[100622]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:38 smithi184 ceph-mon[102844]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:38 smithi196 ceph-mon[76349]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:40 smithi196 ceph-mon[76349]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:40 smithi184 ceph-mon[100622]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:40 smithi184 ceph-mon[102844]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:10:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:42.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:42.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:42.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:42 smithi196 ceph-mon[76349]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-mon[102844]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:42 smithi184 ceph-mon[100622]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:44.537 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:44 smithi184 ceph-mon[100622]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:44.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:44.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:44 smithi184 ceph-mon[102844]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:44.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:44 smithi196 ceph-mon[76349]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:46 smithi184 ceph-mon[100622]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:46 smithi184 ceph-mon[102844]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:46 smithi196 ceph-mon[76349]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:48 smithi196 ceph-mon[76349]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:48 smithi184 ceph-mon[100622]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:48 smithi184 ceph-mon[102844]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:10:49.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:50.622 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:50 smithi196 ceph-mon[76349]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:50 smithi184 ceph-mon[100622]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:50 smithi184 ceph-mon[102844]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:51.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:51 smithi196 ceph-mon[76349]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:51.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.485 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[100622]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[102844]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:51.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:10:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:10:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:52.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:52.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:10:52.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:10:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:10:52.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:10:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:53.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:10:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:54 smithi196 ceph-mon[76349]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:54 smithi184 ceph-mon[100622]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:54 smithi184 ceph-mon[102844]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:56 smithi196 ceph-mon[76349]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:56 smithi184 ceph-mon[100622]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:56 smithi184 ceph-mon[102844]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:58.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:58 smithi196 ceph-mon[76349]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:58 smithi184 ceph-mon[100622]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:58 smithi184 ceph-mon[102844]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:10:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:10:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:10:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:00 smithi196 ceph-mon[76349]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:00.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:00 smithi184 ceph-mon[100622]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:00 smithi184 ceph-mon[102844]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:02.261 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-mon[100622]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:02.262 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:11:02.262 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-mon[102844]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:02.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:02.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:02.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:02.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:02.262 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:02.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:02.263 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:11:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:02 smithi196 ceph-mon[76349]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 7h - - 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 7h - - 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 12s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 10s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:11:02.788 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 12s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 10s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 12s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 12s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 10s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 12s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 12s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 10s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 12s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 12s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 12s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 12s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 10s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:11:02.789 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 10s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 10s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 10s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 10s ago 7h 65.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 12s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:11:02.790 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 10s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:11:03.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:11:03.099 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:11:03.373 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:11:03.373 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:11:03.374 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:11:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:03 smithi184 ceph-mon[100622]: from='client.83807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:03.375 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1608343252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:03.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:03 smithi184 ceph-mon[102844]: from='client.83807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:03.375 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1608343252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:03 smithi196 ceph-mon[76349]: from='client.83807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1608343252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:03.696 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:11:03.696 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:03.696 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:03.696 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:04 smithi196 ceph-mon[76349]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:04 smithi196 ceph-mon[76349]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:04 smithi196 ceph-mon[76349]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:04 smithi196 ceph-mon[76349]: from='client.83831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:04 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3101165019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[100622]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[100622]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[100622]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[100622]: from='client.83831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3101165019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[102844]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[102844]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[102844]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[102844]: from='client.83831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:04.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:04 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3101165019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:06.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:06 smithi184 ceph-mon[100622]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:06.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:06 smithi184 ceph-mon[102844]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:06 smithi196 ceph-mon[76349]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:08 smithi184 ceph-mon[100622]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:08 smithi184 ceph-mon[102844]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:08 smithi196 ceph-mon[76349]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:09.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:09.617 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:10.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:10 smithi196 ceph-mon[76349]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:10 smithi184 ceph-mon[100622]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:10 smithi184 ceph-mon[102844]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:11 smithi196 ceph-mon[76349]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[100622]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[102844]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:11:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:12.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:12.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:12.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:13.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:14 smithi184 ceph-mon[100622]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:14 smithi184 ceph-mon[102844]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:14 smithi196 ceph-mon[76349]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:16 smithi184 ceph-mon[100622]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:16 smithi184 ceph-mon[102844]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:16 smithi196 ceph-mon[76349]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:18 smithi184 ceph-mon[100622]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:18 smithi184 ceph-mon[102844]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:18 smithi196 ceph-mon[76349]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:20 smithi184 ceph-mon[100622]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:20 smithi184 ceph-mon[102844]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:20 smithi196 ceph-mon[76349]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-mon[100622]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-mon[102844]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:11:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:22.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:22.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:22.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:22.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:22 smithi196 ceph-mon[76349]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:24 smithi184 ceph-mon[100622]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:24 smithi184 ceph-mon[102844]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:24 smithi196 ceph-mon[76349]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:26.529 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:26 smithi184 ceph-mon[100622]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:26.529 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:26 smithi184 ceph-mon[102844]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:26 smithi196 ceph-mon[76349]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:28 smithi184 ceph-mon[100622]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:28 smithi184 ceph-mon[102844]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:28 smithi196 ceph-mon[76349]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:30 smithi184 ceph-mon[100622]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:30 smithi184 ceph-mon[102844]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:30.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:30 smithi196 ceph-mon[76349]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[100622]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[102844]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:31 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:31 smithi196 ceph-mon[76349]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:31 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:11:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:32.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:32.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:32.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:33.980 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 7h - - 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 7h - - 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 3s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 2s ago 7h 96.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 3s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 2s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 3s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 3s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 3s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 3s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:11:34.510 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 2s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 3s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 3s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 3s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 3s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 2s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 2s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 2s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 2s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 2s ago 7h 65.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 3s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:11:34.511 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 2s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:11:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:34 smithi196 ceph-mon[76349]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:34 smithi184 ceph-mon[100622]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:34 smithi184 ceph-mon[102844]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:11:34.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:11:34.818 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:11:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:11:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:11:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:11:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:11:35.086 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:11:35.399 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:11:35.400 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:35.400 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:35 smithi196 ceph-mon[76349]: from='client.73881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:35 smithi196 ceph-mon[76349]: from='client.83849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3957912447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[100622]: from='client.73881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[100622]: from='client.83849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3957912447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[102844]: from='client.73881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[102844]: from='client.83849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3957912447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:36 smithi196 ceph-mon[76349]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:36 smithi196 ceph-mon[76349]: from='client.83855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:36 smithi196 ceph-mon[76349]: from='client.83867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:36 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1082381762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[100622]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[100622]: from='client.83855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[100622]: from='client.83867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1082381762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[102844]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[102844]: from='client.83855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[102844]: from='client.83867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:36 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1082381762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:38 smithi184 ceph-mon[100622]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:38 smithi184 ceph-mon[102844]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:38 smithi196 ceph-mon[76349]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:40 smithi196 ceph-mon[76349]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:40 smithi184 ceph-mon[100622]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:40 smithi184 ceph-mon[102844]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:42.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:42.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:42.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:42 smithi196 ceph-mon[76349]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-mon[100622]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:42 smithi184 ceph-mon[102844]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:43 smithi184 ceph-mon[100622]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:43 smithi184 ceph-mon[102844]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:43 smithi196 ceph-mon[76349]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:45.785 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:45 smithi184 ceph-mon[100622]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:45.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:45 smithi184 ceph-mon[102844]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:45.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:45 smithi196 ceph-mon[76349]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:47.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:47 smithi184 ceph-mon[100622]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:47.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:47 smithi184 ceph-mon[102844]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:47.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:47 smithi196 ceph-mon[76349]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:48.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:49 smithi184 ceph-mon[100622]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:49 smithi184 ceph-mon[102844]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:49.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:49 smithi196 ceph-mon[76349]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:52.097 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:11:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[100622]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[102844]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.098 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:51 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:51 smithi196 ceph-mon[76349]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:52.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.102 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:51 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:52.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:52.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:11:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:11:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:11:52.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:53 smithi196 ceph-mon[76349]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[102844]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[100622]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:11:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:55 smithi184 ceph-mon[100622]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:55 smithi184 ceph-mon[102844]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:55 smithi196 ceph-mon[76349]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:57 smithi184 ceph-mon[100622]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:57.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:57 smithi184 ceph-mon[102844]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:57 smithi196 ceph-mon[76349]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:59.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:59 smithi196 ceph-mon[76349]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:59.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:11:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:59 smithi184 ceph-mon[100622]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:11:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:59 smithi184 ceph-mon[102844]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:11:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:01 smithi184 ceph-mon[100622]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:01 smithi184 ceph-mon[102844]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:01.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:01 smithi196 ceph-mon[76349]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:12:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:02.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:02.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:02.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:03 smithi184 ceph-mon[100622]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:03 smithi184 ceph-mon[102844]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:03.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:03 smithi196 ceph-mon[76349]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:05.696 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:12:05.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:05 smithi184 ceph-mon[100622]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:05.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:05 smithi184 ceph-mon[102844]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:05 smithi196 ceph-mon[76349]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 7h - - 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 7h - - 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 15s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 13s ago 7h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 15s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 13s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 15s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 15s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 13s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 15s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 15s ago 7h 16.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 13s ago 7h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:12:06.233 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 15s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 15s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 15s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 15s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 13s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 13s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 13s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 13s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 13s ago 7h 67.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 15s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:12:06.234 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 13s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:12:06.553 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:12:06.553 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:12:06.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:12:06.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:12:06.554 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[102844]: from='client.83879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[102844]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[102844]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[100622]: from='client.83879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[100622]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:06 smithi184 ceph-mon[100622]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.825 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:12:06.825 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:12:06.825 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:12:06.826 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:12:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:06 smithi196 ceph-mon[76349]: from='client.83879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:06 smithi196 ceph-mon[76349]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:06 smithi196 ceph-mon[76349]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:07.135 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:12:07.135 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:07.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:07.136 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:07.136 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:07.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[102844]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:07.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2847750595' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:07.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[102844]: from='client.83903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:07.708 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1620755788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:07.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[100622]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:07.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2847750595' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:07.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[100622]: from='client.83903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:07.709 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1620755788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:07.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:07 smithi196 ceph-mon[76349]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2847750595' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:07 smithi196 ceph-mon[76349]: from='client.83903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1620755788' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:08.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:08.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:09 smithi184 ceph-mon[100622]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:09 smithi184 ceph-mon[102844]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:09 smithi196 ceph-mon[76349]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:12.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:11 smithi196 ceph-mon[76349]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:12.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:11 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:12:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[100622]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[102844]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:11 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:12.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:12.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:12.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[100622]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[102844]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:13 smithi196 ceph-mon[76349]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:15.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:15 smithi196 ceph-mon[76349]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:15 smithi184 ceph-mon[100622]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:16.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:15 smithi184 ceph-mon[102844]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:17 smithi196 ceph-mon[76349]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:17 smithi184 ceph-mon[100622]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:17 smithi184 ceph-mon[102844]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:19 smithi196 ceph-mon[76349]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:19 smithi184 ceph-mon[100622]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:19 smithi184 ceph-mon[102844]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:21 smithi196 ceph-mon[76349]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:21 smithi184 ceph-mon[100622]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:21 smithi184 ceph-mon[102844]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:12:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:22.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:22.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:22.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:23 smithi196 ceph-mon[76349]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:23 smithi184 ceph-mon[102844]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:23 smithi184 ceph-mon[100622]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:25 smithi196 ceph-mon[76349]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:25 smithi184 ceph-mon[100622]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:25 smithi184 ceph-mon[102844]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:27 smithi196 ceph-mon[76349]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:27.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:27 smithi184 ceph-mon[102844]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:27.919 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:27 smithi184 ceph-mon[100622]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:28.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:28.739 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:29 smithi196 ceph-mon[76349]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:30.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:29 smithi184 ceph-mon[100622]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:30.021 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:30.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:29 smithi184 ceph-mon[102844]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:30.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[100622]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:12:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[102844]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:32.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:32.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:32.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:32.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:32.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:32.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:32 smithi196 ceph-mon[76349]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:33.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:33 smithi196 ceph-mon[76349]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:33.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:33.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:33.867 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[100622]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[102844]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:34.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:35 smithi184 ceph-mon[100622]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:35 smithi184 ceph-mon[102844]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:35 smithi196 ceph-mon[76349]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 7h - - 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 6s ago 7h 28.7M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 5s ago 7h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 6s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 5s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 6s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 6s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:12:37.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 5s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 6s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 6s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 5s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 6s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 6s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 6s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 5s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 5s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 5s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 5s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 5s ago 7h 67.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:12:37.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:12:37.958 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:12:37.958 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 5s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:12:38.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:37 smithi184 ceph-mon[100622]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:37 smithi184 ceph-mon[102844]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:37 smithi196 ceph-mon[76349]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.267 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:38.268 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:12:38.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:12:38.535 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:12:38.536 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[102844]: from='client.73953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[102844]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[102844]: from='client.73965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1075426765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[100622]: from='client.73953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[100622]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[100622]: from='client.73965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1075426765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:38.846 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:12:38.846 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:38.846 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:38.846 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:38.846 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:38 smithi196 ceph-mon[76349]: from='client.73953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:38 smithi196 ceph-mon[76349]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:38 smithi196 ceph-mon[76349]: from='client.73965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:38 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1075426765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[100622]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[100622]: from='client.83939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1491490983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[102844]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[102844]: from='client.83939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1491490983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:40.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:39 smithi196 ceph-mon[76349]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:39 smithi196 ceph-mon[76349]: from='client.83939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1491490983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:42.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:41 smithi184 ceph-mon[100622]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:42.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:41 smithi184 ceph-mon[102844]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:12:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:41 smithi196 ceph-mon[76349]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:42.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:42.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:42.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:43 smithi184 ceph-mon[100622]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:43 smithi184 ceph-mon[102844]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:43 smithi196 ceph-mon[76349]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:45 smithi184 ceph-mon[100622]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:45 smithi184 ceph-mon[102844]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:45 smithi196 ceph-mon[76349]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:47 smithi184 ceph-mon[100622]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:47 smithi184 ceph-mon[102844]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:47 smithi196 ceph-mon[76349]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.857 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:48 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:48.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:48 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:49.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:48 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:49.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:49 smithi184 ceph-mon[100622]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:49.934 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:49 smithi184 ceph-mon[102844]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:49 smithi196 ceph-mon[76349]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:12:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:12:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[100622]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[102844]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:52.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:52.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:12:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:12:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:12:52.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:12:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:52 smithi196 ceph-mon[76349]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:53 smithi196 ceph-mon[76349]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[100622]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[102844]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:12:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:55 smithi196 ceph-mon[76349]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:55 smithi184 ceph-mon[100622]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:55 smithi184 ceph-mon[102844]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:57 smithi196 ceph-mon[76349]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:58.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:57 smithi184 ceph-mon[100622]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:58.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:57 smithi184 ceph-mon[102844]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:59.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:59.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:59 smithi196 ceph-mon[76349]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:12:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:59 smithi184 ceph-mon[100622]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:12:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:59 smithi184 ceph-mon[102844]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:12:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:01 smithi196 ceph-mon[76349]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:01 smithi184 ceph-mon[100622]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:13:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:01 smithi184 ceph-mon[102844]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:02.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:02.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:02.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:03 smithi184 ceph-mon[100622]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:04.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:03 smithi184 ceph-mon[102844]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:03 smithi196 ceph-mon[76349]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:05 smithi196 ceph-mon[76349]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:05 smithi184 ceph-mon[100622]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:05 smithi184 ceph-mon[102844]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:08.205 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:07 smithi184 ceph-mon[100622]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:08.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:07 smithi184 ceph-mon[102844]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:07 smithi196 ceph-mon[76349]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:09.153 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:13:09.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:09.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:09.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 7h - - 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 7h - - 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 18s ago 7h 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 16s ago 7h 95.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 18s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 16s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 18s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 18s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 16s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:13:09.700 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 18s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 18s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 16s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 18s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 18s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 18s ago 7h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 18s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 16s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 16s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 16s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 16s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 16s ago 7h 69.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:13:09.701 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 18s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:13:09.702 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 16s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:13:09.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:09 smithi184 ceph-mon[100622]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:09.960 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:09 smithi184 ceph-mon[100622]: from='client.73989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:09.960 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:09 smithi184 ceph-mon[102844]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:09.961 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:09 smithi184 ceph-mon[102844]: from='client.73989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:10.020 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:13:10.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:13:10.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:13:10.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:13:10.330 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:13:10.331 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:13:10.332 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:13:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:09 smithi196 ceph-mon[76349]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:09 smithi196 ceph-mon[76349]: from='client.73989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:10.654 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:13:10.654 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:10.654 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:10.654 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:10.655 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[100622]: from='client.73995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[100622]: from='client.83963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1536924840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[100622]: from='client.83975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1752349296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[102844]: from='client.73995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[102844]: from='client.83963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1536924840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[102844]: from='client.83975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1752349296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:11.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:10 smithi196 ceph-mon[76349]: from='client.73995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:10 smithi196 ceph-mon[76349]: from='client.83963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1536924840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:10 smithi196 ceph-mon[76349]: from='client.83975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:11.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1752349296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:13:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:12.101Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:12.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:12.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:12 smithi196 ceph-mon[76349]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[100622]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[102844]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:14 smithi196 ceph-mon[76349]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:14.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[100622]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[102844]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:15.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:16 smithi184 ceph-mon[100622]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:16 smithi184 ceph-mon[102844]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:16.355 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:16 smithi196 ceph-mon[76349]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:18 smithi184 ceph-mon[100622]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:18 smithi184 ceph-mon[102844]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:18 smithi196 ceph-mon[76349]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:19.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:20 smithi184 ceph-mon[100622]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:20 smithi184 ceph-mon[102844]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:20 smithi196 ceph-mon[76349]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-mon[100622]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-mon[102844]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:13:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:22.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:22.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:22.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:22.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:22 smithi196 ceph-mon[76349]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:24 smithi184 ceph-mon[100622]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:24 smithi184 ceph-mon[102844]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:24 smithi196 ceph-mon[76349]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:26 smithi184 ceph-mon[100622]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:26 smithi184 ceph-mon[102844]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:26 smithi196 ceph-mon[76349]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:28 smithi184 ceph-mon[100622]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:28 smithi184 ceph-mon[102844]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:28.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:28 smithi196 ceph-mon[76349]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:29.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:29.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:30.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:30 smithi184 ceph-mon[100622]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:30.402 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:30 smithi184 ceph-mon[102844]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:30.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:30 smithi196 ceph-mon[76349]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:32.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:13:32.306 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:32.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:32.306 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:32.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:32.306 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:32.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:32 smithi196 ceph-mon[76349]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[100622]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[102844]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:34 smithi196 ceph-mon[76349]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[100622]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[102844]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:36 smithi196 ceph-mon[76349]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:36 smithi184 ceph-mon[100622]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:36.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:36 smithi184 ceph-mon[102844]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:38.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:38 smithi196 ceph-mon[76349]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:38 smithi184 ceph-mon[100622]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:38 smithi184 ceph-mon[102844]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:40 smithi184 ceph-mon[100622]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:40 smithi184 ceph-mon[102844]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:40 smithi196 ceph-mon[76349]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:40.949 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:13:41.478 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:13:41.478 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 7h - - 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 10s ago 7h 28.8M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 8s ago 7h 95.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 10s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 8s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 10s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 8s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 10s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 10s ago 7h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 8s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 10s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 10s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:13:41.479 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 10s ago 7h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 10s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 8s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 8s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 8s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 8s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 8s ago 7h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:13:41.480 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 8s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:13:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:13:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:13:41.789 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:13:42.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:13:42.055 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:13:42.056 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:13:42.056 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:13:42.368 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:13:42.368 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:42.368 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:42.368 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:42.369 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[100622]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[100622]: from='client.74025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[100622]: from='client.83993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/707280540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[102844]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:42.369 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[102844]: from='client.74025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[102844]: from='client.83993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.370 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/707280540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:42.370 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:42.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:42.370 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:42.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:42.370 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:42.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:42 smithi196 ceph-mon[76349]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:42 smithi196 ceph-mon[76349]: from='client.74025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:42 smithi196 ceph-mon[76349]: from='client.83993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/707280540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[100622]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[100622]: from='client.84011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2035844722' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:43.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[102844]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[102844]: from='client.84011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:43 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2035844722' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:43 smithi196 ceph-mon[76349]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:43 smithi196 ceph-mon[76349]: from='client.84011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:43.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:43 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2035844722' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:44 smithi196 ceph-mon[76349]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:44 smithi184 ceph-mon[100622]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:44 smithi184 ceph-mon[102844]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:46 smithi196 ceph-mon[76349]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:46 smithi184 ceph-mon[100622]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:46 smithi184 ceph-mon[102844]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:48 smithi184 ceph-mon[102844]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:48 smithi184 ceph-mon[100622]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:48 smithi196 ceph-mon[76349]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:50 smithi184 ceph-mon[100622]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:50 smithi184 ceph-mon[102844]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:50 smithi196 ceph-mon[76349]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:52.104 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:13:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:13:52.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[100622]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:52.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[102844]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.512 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:52.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:52.512 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:52.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:13:52.512 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:13:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:13:52.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:13:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:52 smithi196 ceph-mon[76349]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[100622]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[102844]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:54 smithi196 ceph-mon[76349]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:13:56.545 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:56 smithi196 ceph-mon[76349]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:56 smithi184 ceph-mon[100622]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:56 smithi184 ceph-mon[102844]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:58 smithi184 ceph-mon[100622]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:58 smithi184 ceph-mon[102844]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:58 smithi196 ceph-mon[76349]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:13:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:13:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:59.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:13:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:00 smithi184 ceph-mon[100622]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:00 smithi184 ceph-mon[102844]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:00 smithi196 ceph-mon[76349]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:02.272 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:14:02.272 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:02.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:02.272 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:02.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:02.273 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:02.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-mon[100622]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:02 smithi184 ceph-mon[102844]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:02 smithi196 ceph-mon[76349]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:04 smithi184 ceph-mon[100622]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:04 smithi184 ceph-mon[102844]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:04 smithi196 ceph-mon[76349]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:06 smithi184 ceph-mon[100622]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:06 smithi184 ceph-mon[102844]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:06 smithi196 ceph-mon[76349]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:08 smithi184 ceph-mon[100622]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:08 smithi184 ceph-mon[102844]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:08 smithi196 ceph-mon[76349]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:09.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:09.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:10 smithi184 ceph-mon[100622]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:10 smithi184 ceph-mon[102844]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:10 smithi196 ceph-mon[76349]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:14:12.231 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:14:12.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:12.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:12.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:12.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:12.231 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:12.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[100622]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[102844]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:12.511 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:12 smithi196 ceph-mon[76349]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:12.670 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 7h - - 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 7h - - 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 1s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 19s ago 7h 94.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 1s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 19s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 1s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 1s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 19s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 1s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 1s ago 7h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:14:13.209 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 19s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 1s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 1s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 1s ago 7h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 1s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 19s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 19s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 19s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 19s ago 7h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 1s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:14:13.210 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 19s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:14:13.543 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:14:13.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:13.545 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:14:13.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:14:13.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:14:13.545 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:14:13.813 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:14:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:14:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:14:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:14:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:14:13.814 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:14:14.123 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:14:14.123 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:14.123 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='client.74061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3167541453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3603677098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='client.74061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3167541453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3603677098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:14.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='client.74061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3167541453' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3603677098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:15.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:15 smithi196 ceph-mon[76349]: from='client.84047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:15.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:15 smithi184 ceph-mon[100622]: from='client.84047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:15.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:15 smithi184 ceph-mon[102844]: from='client.84047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:16.681 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:16 smithi196 ceph-mon[76349]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:14:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:16 smithi184 ceph-mon[100622]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:14:16.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:16 smithi184 ceph-mon[102844]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:14:17.767 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:17 smithi196 ceph-mon[76349]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:14:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:17 smithi184 ceph-mon[100622]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:14:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:17 smithi184 ceph-mon[102844]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:14:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:18 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:18.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:18 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:18 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:19 smithi184 ceph-mon[100622]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T03:14:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:19 smithi184 ceph-mon[102844]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T03:14:19.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:19 smithi196 ceph-mon[76349]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T03:14:21.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:21 smithi196 ceph-mon[76349]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:14:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:21 smithi184 ceph-mon[100622]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:14:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:21 smithi184 ceph-mon[102844]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:14:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:22.105Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:22.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:22.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:23 smithi196 ceph-mon[76349]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-09-21T03:14:24.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:23 smithi184 ceph-mon[100622]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-09-21T03:14:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:23 smithi184 ceph-mon[102844]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-09-21T03:14:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:25 smithi196 ceph-mon[76349]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:14:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:25 smithi184 ceph-mon[100622]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:14:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:25 smithi184 ceph-mon[102844]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:14:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:27 smithi196 ceph-mon[76349]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T03:14:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:27 smithi184 ceph-mon[100622]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T03:14:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:27 smithi184 ceph-mon[102844]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-21T03:14:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:29.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:29.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:29 smithi196 ceph-mon[76349]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T03:14:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:29 smithi184 ceph-mon[100622]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T03:14:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:29 smithi184 ceph-mon[102844]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-21T03:14:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:14:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:32.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:32.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:32.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[100622]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[102844]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.568 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:32 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:32 smithi196 ceph-mon[76349]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 411 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:14:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:32 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:34 smithi196 ceph-mon[76349]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:14:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[100622]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[102844]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:36.606 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:36 smithi196 ceph-mon[76349]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:14:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:36 smithi184 ceph-mon[100622]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:14:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:36 smithi184 ceph-mon[102844]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:14:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:38 smithi196 ceph-mon[76349]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:14:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:38 smithi184 ceph-mon[100622]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:14:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:38 smithi184 ceph-mon[102844]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:14:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:40 smithi196 ceph-mon[76349]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T03:14:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:40 smithi184 ceph-mon[100622]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T03:14:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:40 smithi184 ceph-mon[102844]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T03:14:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:42.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:42.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:42.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:42 smithi196 ceph-mon[76349]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:14:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-mon[100622]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:14:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:42 smithi184 ceph-mon[102844]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:14:44.420 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:14:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:44 smithi196 ceph-mon[76349]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:14:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:44 smithi184 ceph-mon[100622]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:14:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:44 smithi184 ceph-mon[102844]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:14:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:14:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 7h - - 2024-09-21T03:14:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T03:14:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 13s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:14:44.955 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 11s ago 7h 93.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 13s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 11s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 13s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 13s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 11s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 13s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 13s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 11s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 13s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 13s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 13s ago 7h 86.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:14:44.956 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 13s ago 7h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 11s ago 7h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 11s ago 7h 96.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 11s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 11s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 11s ago 7h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 13s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:14:44.957 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 11s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:14:45.266 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:14:45.266 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:14:45.266 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:14:45.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:14:45.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:14:45.538 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:14:45.538 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:14:45.538 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:14:45.538 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:14:45.538 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:14:45.539 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:14:45.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:45 smithi184 ceph-mon[100622]: from='client.84059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:45.539 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3598619972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:45.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:45 smithi184 ceph-mon[102844]: from='client.84059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:45.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3598619972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:45.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:45 smithi196 ceph-mon[76349]: from='client.84059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3598619972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:45.847 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:14:45.848 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:45.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:45.848 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:45.848 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:46 smithi196 ceph-mon[76349]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:14:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:46 smithi196 ceph-mon[76349]: from='client.74103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:46 smithi196 ceph-mon[76349]: from='client.84071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3545075866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[100622]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:14:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[100622]: from='client.74103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[100622]: from='client.84071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3545075866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[102844]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:14:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[102844]: from='client.74103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[102844]: from='client.84071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3545075866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:47.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:47 smithi196 ceph-mon[76349]: from='client.84083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:47.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:47 smithi184 ceph-mon[100622]: from='client.84083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:47.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:47 smithi184 ceph-mon[102844]: from='client.84083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:48 smithi196 ceph-mon[76349]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:14:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:48 smithi184 ceph-mon[100622]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:14:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:48 smithi184 ceph-mon[102844]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:14:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:50 smithi196 ceph-mon[76349]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:14:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:50 smithi184 ceph-mon[100622]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:14:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:50 smithi184 ceph-mon[102844]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:14:52.235 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:14:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:14:52.235 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:52.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:52.235 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:52.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:14:52.235 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:14:52.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:14:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[100622]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:14:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[102844]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:14:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:52 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:52 smithi196 ceph-mon[76349]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:14:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:52 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[100622]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[102844]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:54 smithi196 ceph-mon[76349]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-21T03:14:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:14:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:56 smithi196 ceph-mon[76349]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:14:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:56 smithi184 ceph-mon[100622]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:14:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:56 smithi184 ceph-mon[102844]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:14:58.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:58 smithi196 ceph-mon[76349]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:58 smithi184 ceph-mon[102844]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:58 smithi184 ceph-mon[100622]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:59.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:59.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:14:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:14:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:14:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:00 smithi196 ceph-mon[76349]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:00 smithi184 ceph-mon[100622]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:00 smithi184 ceph-mon[102844]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:02.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:02.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:02.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:02.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:02 smithi196 ceph-mon[76349]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-mon[100622]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:02 smithi184 ceph-mon[102844]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:04 smithi196 ceph-mon[76349]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:04 smithi184 ceph-mon[100622]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:04 smithi184 ceph-mon[102844]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:06 smithi196 ceph-mon[76349]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:06 smithi184 ceph-mon[100622]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:06 smithi184 ceph-mon[102844]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:08 smithi184 ceph-mon[100622]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:08 smithi184 ceph-mon[102844]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:08 smithi196 ceph-mon[76349]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:10 smithi184 ceph-mon[100622]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:10 smithi184 ceph-mon[102844]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:10.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:10 smithi196 ceph-mon[76349]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:15:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:12.108Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:12.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:12.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:12.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[102844]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.727 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[100622]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.728 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:12 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:12 smithi196 ceph-mon[76349]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:12 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[100622]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[102844]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:14 smithi196 ceph-mon[76349]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:16.143 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:15:16.703 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:16 smithi184 ceph-mon[100622]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:16.704 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:16 smithi184 ceph-mon[102844]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 7h - - 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 7h - - 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 4s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 2s ago 7h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:15:16.704 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 4s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 2s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 4s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 4s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 4s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 4s ago 7h 16.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 2s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 4s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 4s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 4s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 4s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 2s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 2s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:15:16.705 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 2s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 2s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 2s ago 7h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 4s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:15:16.706 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 2s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:15:16.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:16 smithi196 ceph-mon[76349]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:17.023 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:15:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:15:17.024 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:15:17.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:15:17.293 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:15:17.293 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:15:17.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[100622]: from='client.84095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[100622]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4249075841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[102844]: from='client.84095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[102844]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4249075841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:17.612 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:15:17.612 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:17.612 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:17 smithi196 ceph-mon[76349]: from='client.84095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:17 smithi196 ceph-mon[76349]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:17.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4249075841' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:18.556 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:18 smithi196 ceph-mon[76349]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:18 smithi196 ceph-mon[76349]: from='client.84107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:18 smithi196 ceph-mon[76349]: from='client.84119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.557 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:18 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/272138670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[100622]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[100622]: from='client.84107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[100622]: from='client.84119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/272138670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[102844]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[102844]: from='client.84107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[102844]: from='client.84119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:18 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/272138670' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:19.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:19.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:20.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:20 smithi196 ceph-mon[76349]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:20 smithi184 ceph-mon[100622]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:20 smithi184 ceph-mon[102844]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:22.108Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:22.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:22.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-mon[100622]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:22 smithi184 ceph-mon[102844]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:22 smithi196 ceph-mon[76349]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:23.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:23 smithi184 ceph-mon[100622]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:23.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:23 smithi184 ceph-mon[102844]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:23.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:23 smithi196 ceph-mon[76349]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:25.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:25 smithi196 ceph-mon[76349]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:25 smithi184 ceph-mon[100622]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:25 smithi184 ceph-mon[102844]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:27 smithi196 ceph-mon[76349]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:27 smithi184 ceph-mon[100622]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:27 smithi184 ceph-mon[102844]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:28 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:28 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:28 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:29 smithi196 ceph-mon[76349]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:29 smithi184 ceph-mon[100622]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:30.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:29 smithi184 ceph-mon[102844]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:31.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:31 smithi196 ceph-mon[76349]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:31 smithi184 ceph-mon[100622]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:31 smithi184 ceph-mon[102844]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:15:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:32.109Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:32.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:32.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.540 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.541 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:33.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[100622]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[102844]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.494 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:34 smithi196 ceph-mon[76349]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:36.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:36 smithi184 ceph-mon[100622]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:36 smithi184 ceph-mon[102844]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:36 smithi196 ceph-mon[76349]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:38 smithi184 ceph-mon[100622]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:38 smithi184 ceph-mon[102844]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:38.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:38 smithi196 ceph-mon[76349]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:39.596 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:40 smithi184 ceph-mon[100622]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:40 smithi184 ceph-mon[102844]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:40 smithi196 ceph-mon[76349]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-mon[100622]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-mon[102844]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:15:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:42.110Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:42.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:42.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:42 smithi196 ceph-mon[76349]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:44 smithi184 ceph-mon[100622]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:44 smithi184 ceph-mon[102844]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:44 smithi196 ceph-mon[76349]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:46 smithi184 ceph-mon[100622]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:46 smithi184 ceph-mon[102844]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:46 smithi196 ceph-mon[76349]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:47.906 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 16s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:15:48.445 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 16s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 14s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 16s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 16s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 14s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 16s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 16s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 14s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:15:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 16s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 16s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 16s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 16s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 14s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 14s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 14s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:15:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 14s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:15:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 14s ago 7h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:15:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:15:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:15:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:15:48.448 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 14s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:15:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:48 smithi184 ceph-mon[100622]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:48 smithi184 ceph-mon[102844]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:48 smithi196 ceph-mon[76349]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:48.754 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:15:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:15:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:15:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:15:48.755 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:15:49.026 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:15:49.027 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[102844]: from='client.84131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[102844]: from='client.84137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1701805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[100622]: from='client.84131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[100622]: from='client.84137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:49.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1701805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:15:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:49.338 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:49.338 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:49.338 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:49 smithi196 ceph-mon[76349]: from='client.84131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:49 smithi196 ceph-mon[76349]: from='client.84137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:49.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:49.728 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1701805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[100622]: from='client.84143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[100622]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[100622]: from='client.84155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/862994546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[102844]: from='client.84143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[102844]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[102844]: from='client.84155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:50 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/862994546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:50 smithi196 ceph-mon[76349]: from='client.84143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:50 smithi196 ceph-mon[76349]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:50 smithi196 ceph-mon[76349]: from='client.84155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:50 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/862994546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:52.257 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:15:52] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:15:52.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:52.111Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:52.257 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:52.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:15:52.258 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:15:52.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:15:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-mon[100622]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:52 smithi184 ceph-mon[102844]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:52 smithi196 ceph-mon[76349]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:53.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[102844]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:54.512 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[100622]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.513 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:54 smithi196 ceph-mon[76349]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:15:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:56 smithi184 ceph-mon[100622]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:56 smithi184 ceph-mon[102844]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:56 smithi196 ceph-mon[76349]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:58 smithi196 ceph-mon[76349]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:58 smithi184 ceph-mon[100622]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:58 smithi184 ceph-mon[102844]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:15:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:15:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:15:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:00.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:00 smithi196 ceph-mon[76349]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:00 smithi184 ceph-mon[100622]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:00 smithi184 ceph-mon[102844]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:16:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:02.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:02.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:02.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:02 smithi196 ceph-mon[76349]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-mon[100622]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:02 smithi184 ceph-mon[102844]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:04.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:04 smithi196 ceph-mon[76349]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:04 smithi184 ceph-mon[100622]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:04 smithi184 ceph-mon[102844]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:06.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:06 smithi196 ceph-mon[76349]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:06.710 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:06 smithi184 ceph-mon[100622]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:06.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:06 smithi184 ceph-mon[102844]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:08.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:08 smithi196 ceph-mon[76349]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:08 smithi184 ceph-mon[100622]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:08 smithi184 ceph-mon[102844]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:09.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:09.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:10 smithi184 ceph-mon[100622]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:10 smithi184 ceph-mon[102844]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:10 smithi196 ceph-mon[76349]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:16:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:12.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:12.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:12.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-mon[100622]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:12 smithi184 ceph-mon[102844]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:12.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:12 smithi196 ceph-mon[76349]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[100622]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:13 smithi184 ceph-mon[102844]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:13.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:13 smithi196 ceph-mon[76349]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:15.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:16.435 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:16 smithi184 ceph-mon[100622]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:16.435 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:16 smithi184 ceph-mon[102844]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:16 smithi196 ceph-mon[76349]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:18 smithi184 ceph-mon[100622]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:18 smithi184 ceph-mon[102844]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:18 smithi196 ceph-mon[76349]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:19.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:19.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:19.637 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:16:19.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 7h - - 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 7s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 5s ago 7h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 7s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 5s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 7s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 7s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:16:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 5s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 7s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 7s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 5s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 7s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 7s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 7s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 7s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 5s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 5s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 5s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 5s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 5s ago 7h 70.4M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:16:20.164 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:16:20.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 7s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:16:20.165 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 5s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:16:20.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:20 smithi196 ceph-mon[76349]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.476 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:16:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:16:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:16:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:16:20.477 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:16:20.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:20 smithi184 ceph-mon[100622]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:20 smithi184 ceph-mon[102844]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.747 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:16:20.747 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:16:20.748 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:16:21.058 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:16:21.312 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:21.312 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:21.313 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:21.313 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:21.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:21 smithi196 ceph-mon[76349]: from='client.74205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:21 smithi196 ceph-mon[76349]: from='client.84173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:21 smithi196 ceph-mon[76349]: from='client.74217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/896517344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:21.372 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:21 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1555838913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[100622]: from='client.74205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[100622]: from='client.84173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[100622]: from='client.74217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/896517344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:21.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1555838913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[102844]: from='client.74205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[102844]: from='client.84173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[102844]: from='client.74217 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/896517344' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:21.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:21 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1555838913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-mon[100622]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-mon[100622]: from='client.74229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:16:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-mon[102844]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-mon[102844]: from='client.74229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:22.114Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:22.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:22.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:22 smithi196 ceph-mon[76349]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:22 smithi196 ceph-mon[76349]: from='client.74229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:24 smithi196 ceph-mon[76349]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:24 smithi184 ceph-mon[100622]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:24 smithi184 ceph-mon[102844]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:26 smithi184 ceph-mon[100622]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:26 smithi184 ceph-mon[102844]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:26.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:26 smithi196 ceph-mon[76349]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:28 smithi184 ceph-mon[100622]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:28 smithi184 ceph-mon[102844]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:28 smithi196 ceph-mon[76349]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:29.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:29.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:30 smithi184 ceph-mon[100622]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:30 smithi184 ceph-mon[102844]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:30 smithi196 ceph-mon[76349]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-mon[100622]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-mon[102844]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:16:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:32.115Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:32.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:32.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:32 smithi196 ceph-mon[76349]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:33 smithi196 ceph-mon[76349]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:33.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[102844]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[100622]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:34.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:35.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:36.562 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:36 smithi184 ceph-mon[100622]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:36.562 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:36 smithi184 ceph-mon[102844]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:36 smithi196 ceph-mon[76349]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:38.403 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:38 smithi184 ceph-mon[100622]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:38.404 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:38 smithi184 ceph-mon[102844]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:38.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:38 smithi196 ceph-mon[76349]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:40.620 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:40 smithi196 ceph-mon[76349]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:40 smithi184 ceph-mon[100622]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:40 smithi184 ceph-mon[102844]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:16:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:42.116Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:42.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:42.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:42.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:42 smithi196 ceph-mon[76349]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-mon[100622]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:42 smithi184 ceph-mon[102844]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:44 smithi196 ceph-mon[76349]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:44.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:44 smithi184 ceph-mon[100622]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:44 smithi184 ceph-mon[102844]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:46 smithi196 ceph-mon[76349]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:46 smithi184 ceph-mon[100622]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:46 smithi184 ceph-mon[102844]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:48.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:48 smithi196 ceph-mon[76349]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:48 smithi184 ceph-mon[100622]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:48 smithi184 ceph-mon[102844]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:49.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:49.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:50 smithi196 ceph-mon[76349]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:50 smithi184 ceph-mon[100622]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:50 smithi184 ceph-mon[102844]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:51.352 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:16:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:16:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 7h - - 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 17s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 19s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 17s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 19s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 17s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 19s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 17s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 19s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 19s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 19s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 19s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:16:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 17s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 17s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 17s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 17s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 17s ago 7h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:16:51.880 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 17s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:52.190 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:16:52.191 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:16:52.191 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:16:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:16:52.192 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:52.117Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:52.192 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:52.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:16:52.192 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:16:52.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:16:52.463 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:16:52.464 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:16:52.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[102844]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[102844]: from='client.84203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:52.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2925110066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:52.464 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[100622]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[100622]: from='client.84203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:52.465 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2925110066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:52 smithi196 ceph-mon[76349]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:52 smithi196 ceph-mon[76349]: from='client.84203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2925110066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:52.794 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:16:52.794 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:52.794 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:52.794 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:52.795 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='client.84215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1757898821' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='client.84215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1757898821' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='client.84215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:53.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1757898821' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:54 smithi196 ceph-mon[76349]: from='client.74265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:54.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:54 smithi196 ceph-mon[76349]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:54 smithi184 ceph-mon[100622]: from='client.74265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:54 smithi184 ceph-mon[100622]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:54 smithi184 ceph-mon[102844]: from='client.74265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:54 smithi184 ceph-mon[102844]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:16:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:56 smithi184 ceph-mon[100622]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:56 smithi184 ceph-mon[102844]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:56.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:56 smithi196 ceph-mon[76349]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:57 smithi184 ceph-mon[100622]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:57 smithi184 ceph-mon[102844]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:57.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:57 smithi196 ceph-mon[76349]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:58 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:58 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:58.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:58 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:59.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:59 smithi196 ceph-mon[76349]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:59.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:16:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:59 smithi184 ceph-mon[100622]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:16:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:59 smithi184 ceph-mon[102844]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:16:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:01.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:01 smithi196 ceph-mon[76349]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:01 smithi184 ceph-mon[100622]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:01 smithi184 ceph-mon[102844]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:02.118Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:02.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:02.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:03.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:03 smithi184 ceph-mon[100622]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:03.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:03 smithi184 ceph-mon[102844]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:03.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:03 smithi196 ceph-mon[76349]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:05 smithi196 ceph-mon[76349]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:05 smithi184 ceph-mon[100622]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:05 smithi184 ceph-mon[102844]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:07.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:07 smithi196 ceph-mon[76349]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:07 smithi184 ceph-mon[100622]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:07 smithi184 ceph-mon[102844]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:08 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:08 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:08 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:09.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:09 smithi196 ceph-mon[76349]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:09 smithi184 ceph-mon[100622]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:09 smithi184 ceph-mon[102844]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:11.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:11 smithi196 ceph-mon[76349]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:11 smithi184 ceph-mon[100622]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:11 smithi184 ceph-mon[102844]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:17:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:12.119Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:12.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:12.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[100622]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[102844]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:13 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:13 smithi196 ceph-mon[76349]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:13 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:15 smithi196 ceph-mon[76349]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:15 smithi184 ceph-mon[100622]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:15 smithi184 ceph-mon[102844]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:17 smithi184 ceph-mon[100622]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:18.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:17 smithi184 ceph-mon[102844]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:18.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:17 smithi196 ceph-mon[76349]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:19.950 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:19.951 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:19 smithi184 ceph-mon[100622]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:19.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:19.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:19 smithi184 ceph-mon[102844]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:19 smithi196 ceph-mon[76349]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:22.122 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:21 smithi184 ceph-mon[100622]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:22.122 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:17:22.122 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:21 smithi184 ceph-mon[102844]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:21 smithi196 ceph-mon[76349]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:22.120Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:22.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:22.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:23.093 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 7h - - 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 10s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 9s ago 7h 96.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 10s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 9s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 10s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 9s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 10s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:17:23.619 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 10s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 9s ago 7h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 10s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 10s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 10s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 10s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 9s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 9s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 9s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 9s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 9s ago 7h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:17:23.620 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 9s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:17:23.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[102844]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:23.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[102844]: from='client.74277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:23.927 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[102844]: from='client.84245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:23.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[100622]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:23.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[100622]: from='client.74277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:23.928 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:23 smithi184 ceph-mon[100622]: from='client.84245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:23.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:17:23.929 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:17:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:23 smithi196 ceph-mon[76349]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:24.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:23 smithi196 ceph-mon[76349]: from='client.74277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:23 smithi196 ceph-mon[76349]: from='client.84245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:24.194 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:17:24.194 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:17:24.194 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:17:24.194 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:17:24.195 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:17:24.505 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:17:24.506 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:24.506 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:24.506 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:24.506 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:24 smithi196 ceph-mon[76349]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3421645560' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:24 smithi196 ceph-mon[76349]: from='client.84263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/741119003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[102844]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3421645560' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[102844]: from='client.84263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/741119003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[100622]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3421645560' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[100622]: from='client.84263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:25.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/741119003' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:25 smithi196 ceph-mon[76349]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:25 smithi184 ceph-mon[100622]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:25 smithi184 ceph-mon[102844]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:28.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:27 smithi196 ceph-mon[76349]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:27 smithi184 ceph-mon[100622]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:27 smithi184 ceph-mon[102844]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:30.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:29 smithi196 ceph-mon[76349]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[100622]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[102844]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:31 smithi196 ceph-mon[76349]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:31 smithi184 ceph-mon[100622]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:31 smithi184 ceph-mon[102844]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:17:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:32.121Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:32.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:32.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[100622]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[102844]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:33 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:33 smithi196 ceph-mon[76349]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:33 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[102844]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[100622]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:35 smithi196 ceph-mon[76349]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:37.956 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:37 smithi184 ceph-mon[102844]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:37.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:37 smithi184 ceph-mon[100622]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:38.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:37 smithi196 ceph-mon[76349]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:39 smithi184 ceph-mon[100622]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:39 smithi184 ceph-mon[102844]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:39 smithi196 ceph-mon[76349]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:41 smithi184 ceph-mon[100622]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:41 smithi184 ceph-mon[102844]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:41 smithi196 ceph-mon[76349]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:42.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:42.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:42.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:43.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:43 smithi196 ceph-mon[76349]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:43 smithi184 ceph-mon[100622]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:43 smithi184 ceph-mon[102844]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:45 smithi184 ceph-mon[100622]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:45 smithi184 ceph-mon[102844]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:46.048 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:45 smithi196 ceph-mon[76349]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:47 smithi184 ceph-mon[100622]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:47 smithi184 ceph-mon[102844]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:47 smithi196 ceph-mon[76349]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:49 smithi184 ceph-mon[102844]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:49 smithi184 ceph-mon[100622]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:49 smithi196 ceph-mon[76349]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:51 smithi184 ceph-mon[100622]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:51 smithi184 ceph-mon[102844]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:51 smithi196 ceph-mon[76349]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:17:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:17:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:52.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:52.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:17:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:17:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:17:52.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:17:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:54 smithi196 ceph-mon[76349]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[100622]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[102844]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:54.803 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:17:55.346 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:17:55.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 7h - - 2024-09-21T03:17:55.346 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 7h - - 2024-09-21T03:17:55.346 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 2s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 0s ago 7h 96.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 2s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 0s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 2s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 2s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 2s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 2s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 0s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 2s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 2s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 2s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 2s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 0s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 0s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:17:55.347 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 0s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 0s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 0s ago 7h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:17:55.348 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 0s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:17:55.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:17:55.656 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:17:55.928 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='client.74313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='client.84281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3026375305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='client.74313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.929 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:55.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='client.84281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:55.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3026375305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:56.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='client.74313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='client.84281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3026375305' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:56.238 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:17:56.239 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:56.239 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:56.239 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:56.239 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:56 smithi196 ceph-mon[76349]: from='client.74337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:57.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2729486219' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:56 smithi184 ceph-mon[100622]: from='client.74337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:57.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2729486219' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:56 smithi184 ceph-mon[102844]: from='client.74337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:57.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2729486219' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:57 smithi196 ceph-mon[76349]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:57 smithi184 ceph-mon[100622]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:57 smithi184 ceph-mon[102844]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:00.102 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:00.102 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[100622]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.102 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:00.102 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:00.103 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[102844]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.103 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:17:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:59 smithi196 ceph-mon[76349]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:17:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:02.124 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:01 smithi184 ceph-mon[100622]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:02.124 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:01 smithi184 ceph-mon[102844]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:02.124 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:18:02.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:01 smithi196 ceph-mon[76349]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:02.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:02.123Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:02.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:02.124Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:02.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:02.124Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:03 smithi196 ceph-mon[76349]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:04.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:03 smithi184 ceph-mon[100622]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:03 smithi184 ceph-mon[102844]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:06.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:05 smithi196 ceph-mon[76349]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:05 smithi184 ceph-mon[100622]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:05 smithi184 ceph-mon[102844]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:07 smithi196 ceph-mon[76349]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:07 smithi184 ceph-mon[100622]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:07 smithi184 ceph-mon[102844]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:09 smithi196 ceph-mon[76349]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:09 smithi184 ceph-mon[100622]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:09 smithi184 ceph-mon[102844]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:12.125 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:11 smithi184 ceph-mon[100622]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.125 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:11 smithi184 ceph-mon[102844]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.126 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:18:12.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:11 smithi196 ceph-mon[76349]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:12.124Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:12.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:12.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 03:18:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T03:18:13.812770787Z level=info msg="Completed cleanup jobs" duration=13.328887ms 2024-09-21T03:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[100622]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[102844]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:14 smithi196 ceph-mon[76349]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[100622]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[102844]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:15 smithi196 ceph-mon[76349]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:17 smithi184 ceph-mon[100622]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:17 smithi184 ceph-mon[102844]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:17 smithi196 ceph-mon[76349]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:20.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:20.285 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:19 smithi184 ceph-mon[102844]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:20.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:20.285 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:19 smithi184 ceph-mon[100622]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:20.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:19 smithi196 ceph-mon[76349]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:21 smithi184 ceph-mon[100622]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:21 smithi184 ceph-mon[102844]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:18:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:22.124Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:22.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:22.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:21 smithi196 ceph-mon[76349]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:24.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:23 smithi196 ceph-mon[76349]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:23 smithi184 ceph-mon[100622]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:23 smithi184 ceph-mon[102844]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:25 smithi184 ceph-mon[100622]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:25 smithi184 ceph-mon[102844]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:25 smithi196 ceph-mon[76349]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:26.535 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:18:27.059 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 7h - - 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 7h - - 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 13s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 12s ago 7h 93.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 13s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 12s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 13s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 13s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 12s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 13s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 13s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:18:27.060 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 12s ago 7h 17.8M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 13s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 13s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 13s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 13s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 12s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 12s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 12s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 12s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 12s ago 7h 70.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 13s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:18:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 12s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:18:27.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:18:27.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:18:27.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:18:27.640 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:18:27.640 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:18:27.640 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:18:27.640 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:18:27.641 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:18:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:18:27.949 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:27.950 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: from='client.74349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1986007986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3932543757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: from='client.74349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1986007986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3932543757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:28.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: from='client.74349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1986007986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3932543757' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:29.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:28 smithi196 ceph-mon[76349]: from='client.84335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:29.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:28 smithi184 ceph-mon[100622]: from='client.84335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:29.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:28 smithi184 ceph-mon[102844]: from='client.84335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[100622]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[102844]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:30.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:29 smithi196 ceph-mon[76349]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:18:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:32.125Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:32.126Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:32.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:32 smithi196 ceph-mon[76349]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-mon[100622]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:32 smithi184 ceph-mon[102844]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[100622]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[102844]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:34 smithi196 ceph-mon[76349]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:36 smithi196 ceph-mon[76349]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[100622]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[102844]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:38 smithi196 ceph-mon[76349]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:38.528 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:38 smithi184 ceph-mon[100622]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:38.528 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:38 smithi184 ceph-mon[102844]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:40.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:40.195 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:40 smithi184 ceph-mon[100622]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:40.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:40.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:40 smithi184 ceph-mon[102844]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:40 smithi196 ceph-mon[76349]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-mon[100622]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:42.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-mon[102844]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:18:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:42.126Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:42.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:42.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:42 smithi196 ceph-mon[76349]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:44 smithi196 ceph-mon[76349]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:44 smithi184 ceph-mon[100622]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:44 smithi184 ceph-mon[102844]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:45.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:46.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:46 smithi196 ceph-mon[76349]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:46 smithi184 ceph-mon[100622]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:46 smithi184 ceph-mon[102844]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:48 smithi196 ceph-mon[76349]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.538 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:48 smithi184 ceph-mon[100622]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.538 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:48 smithi184 ceph-mon[102844]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:50 smithi196 ceph-mon[76349]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:50 smithi184 ceph-mon[100622]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:50 smithi184 ceph-mon[102844]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-mon[100622]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:52.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-mon[102844]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:18:52] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:18:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:52.127Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:52.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:18:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:18:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:18:52.128Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:18:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:52 smithi196 ceph-mon[76349]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:54 smithi196 ceph-mon[76349]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[100622]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[102844]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[100622]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[102844]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:56 smithi196 ceph-mon[76349]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:18:58.248 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:18:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:58 smithi184 ceph-mon[100622]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:58 smithi184 ceph-mon[102844]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:58 smithi196 ceph-mon[76349]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.775 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:18:58.775 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 7h - - 2024-09-21T03:18:58.775 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 7h - - 2024-09-21T03:18:58.775 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 5s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:18:58.775 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 3s ago 7h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 5s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 3s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 5s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 5s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 3s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 5s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 5s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 3s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 5s ago 7h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 5s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 5s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 5s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 3s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 3s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:18:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 3s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 3s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 3s ago 7h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:18:58.777 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 3s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:18:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:18:59.084 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:18:59.358 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:18:59.359 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:18:59.360 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:18:59.360 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:18:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:59 smithi184 ceph-mon[100622]: from='client.84347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:59.360 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:18:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3804465856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:59.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:59 smithi184 ceph-mon[102844]: from='client.84347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:59.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:18:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3804465856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:59 smithi196 ceph-mon[76349]: from='client.84347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:59.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:18:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3804465856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:59.666 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:18:59.666 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:59.666 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:59.666 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:59.666 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='client.74391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='client.74397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='client.74409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.485 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3253591181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='client.74391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='client.74397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='client.74409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.486 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:00 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3253591181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:00.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='client.74391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='client.74397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='client.74409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:00.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:00 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3253591181' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:02.129 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:19:02.384 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-mon[100622]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:02.385 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-mon[102844]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:02.385 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:02.127Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:02.385 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:02.128Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:02.385 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:02.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:02 smithi196 ceph-mon[76349]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:04 smithi184 ceph-mon[100622]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:04 smithi184 ceph-mon[102844]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:04.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:04 smithi196 ceph-mon[76349]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:06 smithi184 ceph-mon[100622]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:06 smithi184 ceph-mon[102844]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:06.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:06 smithi196 ceph-mon[76349]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:08 smithi184 ceph-mon[100622]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:08 smithi184 ceph-mon[102844]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:08 smithi196 ceph-mon[76349]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:10 smithi184 ceph-mon[100622]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:10.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:10 smithi184 ceph-mon[102844]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:10 smithi196 ceph-mon[76349]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-mon[102844]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-mon[100622]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:19:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:12.128Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:12.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:12.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:12 smithi196 ceph-mon[76349]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[102844]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[100622]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:14.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:14 smithi196 ceph-mon[76349]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:16 smithi196 ceph-mon[76349]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[100622]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[102844]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:18.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:18 smithi196 ceph-mon[76349]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:18 smithi184 ceph-mon[100622]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:18 smithi184 ceph-mon[102844]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:20 smithi196 ceph-mon[76349]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:20 smithi184 ceph-mon[100622]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:20 smithi184 ceph-mon[102844]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:22.130 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:19:22.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-mon[100622]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:22.380 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-mon[102844]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:22.381 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:22.128Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:22.381 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:22.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:22.381 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:22.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:22 smithi196 ceph-mon[76349]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:24 smithi184 ceph-mon[100622]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:24 smithi184 ceph-mon[102844]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:24 smithi196 ceph-mon[76349]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:26.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:26 smithi196 ceph-mon[76349]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:26 smithi184 ceph-mon[100622]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:26 smithi184 ceph-mon[102844]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:28.460 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:28 smithi196 ceph-mon[76349]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:28 smithi184 ceph-mon[100622]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:28 smithi184 ceph-mon[102844]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:29.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:29.961 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:19:30.493 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:19:30.493 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T03:19:30.493 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 7h - - 2024-09-21T03:19:30.493 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 16s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 15s ago 7h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 16s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 15s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 16s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 16s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 15s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 16s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 16s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 15s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 16s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 16s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 16s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:19:30.494 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 16s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 15s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 15s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 15s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 15s ago 7h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:19:30.495 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 15s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:19:30.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:30.595 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:30 smithi196 ceph-mon[76349]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:30 smithi184 ceph-mon[100622]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:30 smithi184 ceph-mon[102844]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.803 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:19:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:19:30.804 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:19:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:19:31.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:19:31.385 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:19:31.385 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:31.386 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:31.386 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:31.386 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:31.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:31 smithi196 ceph-mon[76349]: from='client.84383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:31 smithi196 ceph-mon[76349]: from='client.84389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/209658588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[100622]: from='client.84383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[100622]: from='client.84389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/209658588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:31.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[102844]: from='client.84383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[102844]: from='client.84389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:31.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/209658588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:19:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:32.129Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:32.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:32.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:32 smithi196 ceph-mon[76349]: from='client.74433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:32 smithi196 ceph-mon[76349]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:32 smithi196 ceph-mon[76349]: from='client.74445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:32 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1979988797' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[100622]: from='client.74433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[100622]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[100622]: from='client.74445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1979988797' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[102844]: from='client.74433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[102844]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[102844]: from='client.74445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:32 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1979988797' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:34.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:34 smithi196 ceph-mon[76349]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[100622]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[102844]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[100622]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[102844]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:36 smithi196 ceph-mon[76349]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:38 smithi184 ceph-mon[100622]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:38 smithi184 ceph-mon[102844]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:38 smithi196 ceph-mon[76349]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:39 smithi184 ceph-mon[100622]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:39.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:39 smithi184 ceph-mon[102844]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:39 smithi196 ceph-mon[76349]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:41.918 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:41 smithi184 ceph-mon[100622]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:41.919 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:41 smithi184 ceph-mon[102844]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:42.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:41 smithi196 ceph-mon[76349]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:19:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:42.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:42.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:42.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:43.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:43 smithi184 ceph-mon[100622]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:43.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:43 smithi184 ceph-mon[102844]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:43 smithi196 ceph-mon[76349]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:45.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:45 smithi184 ceph-mon[100622]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:45 smithi184 ceph-mon[102844]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:46.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:45 smithi196 ceph-mon[76349]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:48.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:47 smithi184 ceph-mon[100622]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:47 smithi184 ceph-mon[102844]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.068 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:47 smithi196 ceph-mon[76349]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:49 smithi196 ceph-mon[76349]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:49 smithi184 ceph-mon[100622]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:49 smithi184 ceph-mon[102844]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:52.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:51 smithi184 ceph-mon[100622]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:52.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:51 smithi184 ceph-mon[102844]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:51 smithi196 ceph-mon[76349]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:19:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:52.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:52.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:19:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:19:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:19:52.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:19:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:53 smithi184 ceph-mon[100622]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:53 smithi184 ceph-mon[102844]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:54.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:53 smithi196 ceph-mon[76349]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:55.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[100622]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[102844]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:55 smithi196 ceph-mon[76349]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:19:58.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:57 smithi184 ceph-mon[102844]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:57 smithi184 ceph-mon[100622]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:57 smithi196 ceph-mon[76349]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:00.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:00.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:59 smithi196 ceph-mon[76349]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:00.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:19:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[102844]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:00.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[100622]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:19:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T03:19:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:01.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:01.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:01.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:01.680 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:20:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:01 smithi184 ceph-mon[100622]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:01 smithi184 ceph-mon[102844]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:02.217 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 7h - - 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 7h - - 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 8s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 6s ago 7h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 8s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 6s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 8s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 8s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 6s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 8s ago 7h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 8s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:20:02.218 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 6s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 8s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 8s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 8s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 8s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 6s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 6s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 6s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 6s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 6s ago 7h 70.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:20:02.219 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 6s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:20:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:20:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:02.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:02.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:02.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:01 smithi196 ceph-mon[76349]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:20:02.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:20:02.547 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:20:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:20:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:20:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:20:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:20:02.839 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:20:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[100622]: from='client.74457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[100622]: from='client.84425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[100622]: from='client.74469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/437037875' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[102844]: from='client.74457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[102844]: from='client.84425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[102844]: from='client.74469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:02 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/437037875' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:03.162 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:03.163 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:03.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:03.163 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:03.163 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:03.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:02 smithi196 ceph-mon[76349]: from='client.74457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:02 smithi196 ceph-mon[76349]: from='client.84425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:02 smithi196 ceph-mon[76349]: from='client.74469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:02 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/437037875' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[100622]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[100622]: from='client.74481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3208433645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[102844]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[102844]: from='client.74481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:04.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3208433645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:04.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:03 smithi196 ceph-mon[76349]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:03 smithi196 ceph-mon[76349]: from='client.74481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3208433645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:05 smithi184 ceph-mon[100622]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:05 smithi184 ceph-mon[102844]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:05 smithi196 ceph-mon[76349]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:08.249 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:07 smithi196 ceph-mon[76349]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:07 smithi184 ceph-mon[100622]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:07 smithi184 ceph-mon[102844]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:10.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:10.107 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:09 smithi196 ceph-mon[76349]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:09 smithi184 ceph-mon[100622]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:09 smithi184 ceph-mon[102844]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:11 smithi184 ceph-mon[100622]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:11 smithi184 ceph-mon[102844]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:12.132Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:12.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:12.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:11 smithi196 ceph-mon[76349]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:13 smithi184 ceph-mon[100622]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:14.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:13 smithi184 ceph-mon[102844]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:14.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:13 smithi196 ceph-mon[76349]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[100622]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[102844]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:16 smithi196 ceph-mon[76349]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:18 smithi184 ceph-mon[100622]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:18.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:18 smithi184 ceph-mon[102844]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:18.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:18 smithi196 ceph-mon[76349]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:20 smithi184 ceph-mon[100622]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:20.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:20 smithi184 ceph-mon[102844]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:20 smithi196 ceph-mon[76349]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-mon[100622]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:22.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-mon[102844]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:20:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:22.133Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:22.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:22.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:22 smithi196 ceph-mon[76349]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:24.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:24 smithi184 ceph-mon[100622]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:24.240 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:24 smithi184 ceph-mon[102844]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:24.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:24 smithi196 ceph-mon[76349]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:26 smithi184 ceph-mon[102844]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:26.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:26 smithi184 ceph-mon[100622]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:26 smithi196 ceph-mon[76349]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:28.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:28 smithi196 ceph-mon[76349]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:28 smithi184 ceph-mon[100622]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:28 smithi184 ceph-mon[102844]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:30.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:30.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:30 smithi196 ceph-mon[76349]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:30.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[102844]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:30.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[100622]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:30.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:20:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:32.134Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:32.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:32.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:32.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-mon[100622]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:32.551 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:32 smithi184 ceph-mon[102844]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:32.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:32 smithi196 ceph-mon[76349]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:33.461 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 7h - - 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 18s ago 7h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 19s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 18s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 19s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 19s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:20:33.988 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 17.7M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 18s ago 7h 17.7M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 19s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 19s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 19s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 19s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 18s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 18s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 18s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 18s ago 7h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:20:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:20:33.990 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 18s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:34.310 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:20:34.311 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:20:34.584 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:20:34.584 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:20:34.584 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:20:34.584 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:20:34.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:20:34.585 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: from='client.84455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/188369225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: from='client.84455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.586 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.587 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/188369225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: from='client.84455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/188369225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:34.891 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:34.892 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:34.892 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:35 smithi196 ceph-mon[76349]: from='client.74499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:35 smithi196 ceph-mon[76349]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/506037034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[100622]: from='client.74499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[100622]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/506037034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[102844]: from='client.74499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[102844]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:35.801 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/506037034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[100622]: from='client.74517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[100622]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[102844]: from='client.74517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[102844]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:36 smithi196 ceph-mon[76349]: from='client.74517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:36 smithi196 ceph-mon[76349]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:38 smithi184 ceph-mon[100622]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:38 smithi184 ceph-mon[102844]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:38.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:38 smithi196 ceph-mon[76349]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:40.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:40 smithi196 ceph-mon[76349]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:40 smithi184 ceph-mon[102844]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:40 smithi184 ceph-mon[100622]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:20:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:42.134Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:42.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:42.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-mon[100622]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:42 smithi184 ceph-mon[102844]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:42 smithi196 ceph-mon[76349]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:44 smithi184 ceph-mon[102844]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:44 smithi184 ceph-mon[100622]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:44.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:44 smithi196 ceph-mon[76349]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:46 smithi184 ceph-mon[100622]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:46 smithi184 ceph-mon[102844]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:46 smithi196 ceph-mon[76349]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:48 smithi184 ceph-mon[100622]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:48 smithi184 ceph-mon[102844]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:48.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:48 smithi196 ceph-mon[76349]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:50 smithi196 ceph-mon[76349]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:50 smithi184 ceph-mon[100622]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:50 smithi184 ceph-mon[102844]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:20:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:20:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:52.135Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:52.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:20:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:20:52.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:20:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-mon[102844]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:52 smithi184 ceph-mon[100622]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:52 smithi196 ceph-mon[76349]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[100622]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[102844]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:54 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:54 smithi196 ceph-mon[76349]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:54 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[100622]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[102844]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:56 smithi196 ceph-mon[76349]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:20:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:58 smithi184 ceph-mon[100622]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:58 smithi184 ceph-mon[102844]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:58 smithi196 ceph-mon[76349]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:20:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:20:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:59.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:20:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:00.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:00 smithi196 ceph-mon[76349]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:00.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:00 smithi184 ceph-mon[100622]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:00 smithi184 ceph-mon[102844]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:21:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:02.135Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:02.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:02.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-mon[100622]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:02 smithi184 ceph-mon[102844]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:02.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:02 smithi196 ceph-mon[76349]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:04.735 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:04 smithi184 ceph-mon[100622]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:04.735 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:04 smithi184 ceph-mon[102844]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:04 smithi196 ceph-mon[76349]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:05.190 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:21:05.734 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:21:05.734 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 7h - - 2024-09-21T03:21:05.734 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 7h - - 2024-09-21T03:21:05.734 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 11s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:21:05.734 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 9s ago 7h 95.4M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 11s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 9s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 11s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 11s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 9s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 11s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 11s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 9s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 11s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 11s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 11s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 11s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 9s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:21:05.735 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 9s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 9s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 9s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 9s ago 7h 70.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:21:05.736 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 9s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:21:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:21:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:21:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:21:06.062 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:21:06.377 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:21:06.378 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:21:06.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[100622]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:06.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[100622]: from='client.84491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[100622]: from='client.84497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1030392484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:06.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[102844]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:06.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[102844]: from='client.84491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[102844]: from='client.84497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.701 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1030392484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:21:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:06.701 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:06.701 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:06.701 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:06.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:06 smithi196 ceph-mon[76349]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:06 smithi196 ceph-mon[76349]: from='client.84491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:06 smithi196 ceph-mon[76349]: from='client.84497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1030392484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[102844]: from='client.84503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[102844]: from='client.84515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2409988174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[100622]: from='client.84503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[100622]: from='client.84515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:07 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2409988174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:07 smithi196 ceph-mon[76349]: from='client.84503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:07 smithi196 ceph-mon[76349]: from='client.84515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:07.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:07 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2409988174' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:08 smithi184 ceph-mon[102844]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:08 smithi184 ceph-mon[100622]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:08.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:08 smithi196 ceph-mon[76349]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:10 smithi184 ceph-mon[100622]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:10 smithi184 ceph-mon[102844]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:10.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:10.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:10 smithi196 ceph-mon[76349]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:10.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:12.136Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:12.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:12.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-mon[100622]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:12.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:12 smithi184 ceph-mon[102844]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:12 smithi196 ceph-mon[76349]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[100622]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[102844]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:14 smithi196 ceph-mon[76349]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[100622]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[102844]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:16.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:16 smithi196 ceph-mon[76349]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:18 smithi184 ceph-mon[100622]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:18.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:18 smithi184 ceph-mon[102844]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:18 smithi196 ceph-mon[76349]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:20 smithi184 ceph-mon[100622]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:20 smithi184 ceph-mon[102844]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:20.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:20 smithi196 ceph-mon[76349]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:22] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:22.137Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:22.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:22.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-mon[100622]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:22 smithi184 ceph-mon[102844]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:22 smithi196 ceph-mon[76349]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:24 smithi184 ceph-mon[100622]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:24 smithi184 ceph-mon[102844]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:24.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:24 smithi196 ceph-mon[76349]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:26.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:26 smithi184 ceph-mon[100622]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:26.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:26 smithi184 ceph-mon[102844]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:26 smithi196 ceph-mon[76349]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:28 smithi184 ceph-mon[100622]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:28 smithi184 ceph-mon[102844]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:28 smithi196 ceph-mon[76349]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[102844]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[100622]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:29.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:29 smithi196 ceph-mon[76349]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:31.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:31 smithi196 ceph-mon[76349]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:31 smithi184 ceph-mon[100622]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:32.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:31 smithi184 ceph-mon[102844]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:21:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:32.137Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:32.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:32.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:33 smithi184 ceph-mon[100622]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:33 smithi184 ceph-mon[102844]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:34.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:33 smithi196 ceph-mon[76349]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[102844]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:35 smithi184 ceph-mon[100622]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:35.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:35 smithi196 ceph-mon[76349]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:36.993 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:21:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 7h - - 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 7h - - 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 2s ago 7h 28.5M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 1s ago 7h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 2s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 1s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:21:37.520 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 2s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 2s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 1s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 2s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 2s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 1s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 2s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 2s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 2s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 2s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 1s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 1s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 1s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 1s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 1s ago 7h 71.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:21:37.521 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:21:37.522 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:21:37.522 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:21:37.522 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 1s ago 7h 91.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:21:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:21:37.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:21:37.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:21:38.104 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:21:38.105 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:21:38.420 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:21:38.420 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:38.420 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:38.420 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:38.420 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:38.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[100622]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:38.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[100622]: from='client.84527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[100622]: from='client.84533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.420 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2842665225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:38.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[102844]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:38.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[102844]: from='client.84527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[102844]: from='client.84533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.421 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2842665225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:38 smithi196 ceph-mon[76349]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:38 smithi196 ceph-mon[76349]: from='client.84527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:38 smithi196 ceph-mon[76349]: from='client.84533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2842665225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[100622]: from='client.84539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[100622]: from='client.84551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2288156784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[102844]: from='client.84539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[102844]: from='client.84551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:39 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2288156784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:39.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:39 smithi196 ceph-mon[76349]: from='client.84539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:39 smithi196 ceph-mon[76349]: from='client.84551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:39 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2288156784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:40 smithi184 ceph-mon[100622]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:40.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:40 smithi184 ceph-mon[102844]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:40 smithi196 ceph-mon[76349]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:42.265 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:21:42.265 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:42.138Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:42.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:42.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:42.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:42.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:42.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-mon[100622]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:42.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:42 smithi184 ceph-mon[102844]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:42.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:42 smithi196 ceph-mon[76349]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:44 smithi184 ceph-mon[100622]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:44 smithi184 ceph-mon[102844]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:44.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:44 smithi196 ceph-mon[76349]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:46 smithi184 ceph-mon[100622]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:46 smithi184 ceph-mon[102844]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:46 smithi196 ceph-mon[76349]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:48.448 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:48 smithi184 ceph-mon[100622]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:48.449 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:48 smithi184 ceph-mon[102844]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:48 smithi196 ceph-mon[76349]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:50 smithi184 ceph-mon[100622]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:50 smithi184 ceph-mon[102844]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:50.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:50 smithi196 ceph-mon[76349]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:52.281 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:21:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:21:52.281 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:52.138Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:52.281 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:52.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:21:52.281 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:21:52.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:21:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:52 smithi196 ceph-mon[76349]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-mon[100622]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:52.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:52 smithi184 ceph-mon[102844]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:54.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:54 smithi196 ceph-mon[76349]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:54 smithi184 ceph-mon[100622]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:54 smithi184 ceph-mon[102844]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[100622]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:56.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[102844]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:55 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:55 smithi196 ceph-mon[76349]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:55 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:21:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:58 smithi196 ceph-mon[76349]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:21:58.691 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:58 smithi184 ceph-mon[100622]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:21:58.692 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:58 smithi184 ceph-mon[102844]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:21:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:21:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:59.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:21:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:21:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:00.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:00 smithi196 ceph-mon[76349]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.791 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:00 smithi184 ceph-mon[100622]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:00 smithi184 ceph-mon[102844]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:22:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:02.139Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:02.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:02.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:02.554 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-mon[100622]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:02.554 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:02 smithi184 ceph-mon[102844]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:02 smithi196 ceph-mon[76349]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:04 smithi184 ceph-mon[100622]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:04 smithi184 ceph-mon[102844]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:04.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:04 smithi196 ceph-mon[76349]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:06.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:06 smithi184 ceph-mon[100622]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:06.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:06 smithi184 ceph-mon[102844]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:06 smithi196 ceph-mon[76349]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:08.540 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:08 smithi184 ceph-mon[102844]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:08.541 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:08 smithi184 ceph-mon[100622]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:08.724 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:22:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:08 smithi196 ceph-mon[76349]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 14s ago 7h - - 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 12s ago 7h - - 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 14s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 12s ago 7h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 14s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 12s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 14s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 14s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 12s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 14s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:22:09.263 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 14s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 12s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 14s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 14s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 14s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 14s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 12s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 12s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 12s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 12s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 12s ago 7h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 14s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:22:09.264 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 12s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:22:09.571 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:22:09.571 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:22:09.571 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:22:09.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:22:09.572 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:22:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:22:09.837 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:22:10.145 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:22:10.145 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:10.145 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:10.145 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:10.145 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.84563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.74607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.74613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2994054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3711752390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.84563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.74607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.74613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2994054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3711752390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.84563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.74607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.74613 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2994054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:10.839 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3711752390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:11.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:11 smithi196 ceph-mon[76349]: from='client.74625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:11.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:11 smithi184 ceph-mon[100622]: from='client.74625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:11.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:11 smithi184 ceph-mon[102844]: from='client.74625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:22:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:12.140Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:12.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:12.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:12.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:12 smithi196 ceph-mon[76349]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-mon[100622]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:12 smithi184 ceph-mon[102844]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:14 smithi184 ceph-mon[100622]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:14 smithi184 ceph-mon[102844]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:14 smithi196 ceph-mon[76349]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[100622]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[102844]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:15 smithi196 ceph-mon[76349]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[100622]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:17 smithi184 ceph-mon[102844]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:17.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:17 smithi196 ceph-mon[76349]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:19 smithi184 ceph-mon[100622]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:19 smithi184 ceph-mon[102844]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:20.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:19 smithi196 ceph-mon[76349]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:22.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:21 smithi184 ceph-mon[100622]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:22.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:21 smithi184 ceph-mon[102844]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:22.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:21 smithi196 ceph-mon[76349]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:22:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:22.140Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:22.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:22.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:24.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:23 smithi184 ceph-mon[102844]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:24.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:23 smithi184 ceph-mon[100622]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:24.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:23 smithi196 ceph-mon[76349]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:25 smithi184 ceph-mon[100622]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:26.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:25 smithi184 ceph-mon[102844]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:26.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:25 smithi196 ceph-mon[76349]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:28.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:27 smithi184 ceph-mon[100622]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:28.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:27 smithi184 ceph-mon[102844]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:27 smithi196 ceph-mon[76349]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[100622]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:30.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[102844]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:30.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:29 smithi196 ceph-mon[76349]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:32] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:32.141Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:32.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:32.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:32 smithi196 ceph-mon[76349]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:32.751 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-mon[100622]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:32.752 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:32 smithi184 ceph-mon[102844]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:34 smithi196 ceph-mon[76349]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:34 smithi184 ceph-mon[100622]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:34 smithi184 ceph-mon[102844]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[100622]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[102844]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:35 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:35 smithi196 ceph-mon[76349]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:35 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:37 smithi196 ceph-mon[76349]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[102844]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:38.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:37 smithi184 ceph-mon[100622]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:39 smithi184 ceph-mon[100622]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:40.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:39 smithi184 ceph-mon[102844]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:39 smithi196 ceph-mon[76349]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:40.443 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:22:40.975 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:22:40.975 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 7h - - 2024-09-21T03:22:40.975 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 4s ago 7h - - 2024-09-21T03:22:40.975 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 6s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:22:40.975 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 4s ago 7h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 6s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 4s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 6s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 6s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 4s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 6s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 6s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 4s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 6s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 6s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 6s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 6s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 4s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 4s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:22:40.976 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 4s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 4s ago 7h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:22:40.977 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 4s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:22:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:40 smithi184 ceph-mon[100622]: from='client.74637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:40 smithi184 ceph-mon[102844]: from='client.74637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:40 smithi196 ceph-mon[76349]: from='client.74637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:22:41.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:22:41.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:22:41.284 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:22:41.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:22:41.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:22:41.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:22:41.551 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:22:41.552 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:22:41.869 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:22:41.869 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:41.869 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:41.869 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:41.870 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[100622]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[100622]: from='client.84605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[100622]: from='client.84611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1190992316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[102844]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[102844]: from='client.84605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[102844]: from='client.84611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:41.870 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1190992316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:41 smithi196 ceph-mon[76349]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:41 smithi196 ceph-mon[76349]: from='client.84605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:41 smithi196 ceph-mon[76349]: from='client.84611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1190992316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:42.143 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:42] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:22:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:42.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:42.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:42.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-mon[100622]: from='client.84623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:43.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1674290325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:43.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-mon[102844]: from='client.84623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:43.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:42 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1674290325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:43.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:42 smithi196 ceph-mon[76349]: from='client.84623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:43.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:42 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1674290325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:44.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:43 smithi184 ceph-mon[100622]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:44.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:43 smithi184 ceph-mon[102844]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:44.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:43 smithi196 ceph-mon[76349]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:45.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:45.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:45 smithi196 ceph-mon[76349]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:46.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:45 smithi184 ceph-mon[100622]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:46.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:45 smithi184 ceph-mon[102844]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:48.046 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:47 smithi184 ceph-mon[100622]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:48.047 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:47 smithi184 ceph-mon[102844]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:48.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:47 smithi196 ceph-mon[76349]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:49 smithi184 ceph-mon[102844]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:50.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:49 smithi184 ceph-mon[100622]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:50.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:50.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:49 smithi196 ceph-mon[76349]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:52.019 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:51 smithi184 ceph-mon[100622]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:52.019 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:22:52] "GET /metrics HTTP/1.1" 200 38941 "" "Prometheus/2.43.0" 2024-09-21T03:22:52.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:51 smithi184 ceph-mon[102844]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:51 smithi196 ceph-mon[76349]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:52.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:52.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:22:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:22:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:22:52.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:22:53.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:53 smithi196 ceph-mon[76349]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:54.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:53 smithi184 ceph-mon[102844]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:54.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:53 smithi184 ceph-mon[100622]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[100622]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[102844]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:56 smithi196 ceph-mon[76349]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[102844]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[100622]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:57 smithi196 ceph-mon[76349]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:22:58.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[100622]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:00.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[102844]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:00.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:22:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:00.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:59 smithi196 ceph-mon[76349]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:00.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:22:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:01 smithi184 ceph-mon[100622]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:01 smithi184 ceph-mon[102844]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:02.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:01 smithi196 ceph-mon[76349]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:23:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:02.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:02.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:02.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:03 smithi184 ceph-mon[100622]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:04.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:03 smithi184 ceph-mon[102844]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:04.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:03 smithi196 ceph-mon[76349]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:05 smithi184 ceph-mon[100622]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:06.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:05 smithi184 ceph-mon[102844]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:05 smithi196 ceph-mon[76349]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:07 smithi184 ceph-mon[100622]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:07 smithi184 ceph-mon[102844]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:07 smithi196 ceph-mon[76349]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:09 smithi184 ceph-mon[100622]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:09 smithi184 ceph-mon[102844]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:10.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:10.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:09 smithi196 ceph-mon[76349]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:11 smithi184 ceph-mon[100622]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:11 smithi184 ceph-mon[102844]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:12.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:23:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:11 smithi196 ceph-mon[76349]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:23:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:12.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:12.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:12.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:12.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:12.698 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:23:12.698 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 17s ago 7h - - 2024-09-21T03:23:12.698 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 15s ago 7h - - 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 17s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 15s ago 7h 95.5M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 17s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 15s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 17s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 17s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 15s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 17s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 17s ago 7h 17.1M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 15s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 17s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 17s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 17s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 17s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:23:12.699 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 15s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 15s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 15s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 15s ago 7h 71.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 17s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:23:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 15s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:23:12.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-mon[100622]: from='client.74673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:12.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-mon[100622]: from='client.74679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:12.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-mon[102844]: from='client.74673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:12.986 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:12 smithi184 ceph-mon[102844]: from='client.74679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:23:13.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:23:13.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:23:13.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:23:13.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:12 smithi196 ceph-mon[76349]: from='client.74673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:13.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:12 smithi196 ceph-mon[76349]: from='client.74679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:23:13.298 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:23:13.299 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:23:13.613 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:23:13.613 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:13.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:13.613 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:13.613 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[102844]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[102844]: from='client.74685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2530530356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[102844]: from='client.84659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3238443981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[100622]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[100622]: from='client.74685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2530530356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[100622]: from='client.84659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3238443981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:13 smithi196 ceph-mon[76349]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:13 smithi196 ceph-mon[76349]: from='client.74685 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2530530356' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:13 smithi196 ceph-mon[76349]: from='client.84659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3238443981' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:14.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[100622]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[102844]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:16 smithi196 ceph-mon[76349]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:17 smithi196 ceph-mon[76349]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[100622]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[102844]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:19 smithi196 ceph-mon[76349]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:19 smithi184 ceph-mon[100622]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:19 smithi184 ceph-mon[102844]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:23:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:22.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:22.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:22.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-mon[100622]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:22 smithi184 ceph-mon[102844]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:22 smithi196 ceph-mon[76349]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:24 smithi184 ceph-mon[100622]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:24 smithi184 ceph-mon[102844]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:24.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:24 smithi196 ceph-mon[76349]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:26 smithi184 ceph-mon[100622]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:26 smithi184 ceph-mon[102844]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:26 smithi196 ceph-mon[76349]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:28.765 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:28 smithi184 ceph-mon[100622]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:28.765 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:28 smithi184 ceph-mon[102844]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:28.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:28 smithi196 ceph-mon[76349]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:29.627 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:29.627 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:29.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:30.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:30 smithi184 ceph-mon[100622]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:30.774 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:30.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:30 smithi184 ceph-mon[102844]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:30.774 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:30 smithi196 ceph-mon[76349]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:32.147 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:32] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T03:23:32.434 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:32.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:32.434 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:32.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:32.434 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:32.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-mon[100622]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:32 smithi184 ceph-mon[102844]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:32.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:32 smithi196 ceph-mon[76349]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:34 smithi196 ceph-mon[76349]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:34 smithi184 ceph-mon[100622]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:34 smithi184 ceph-mon[102844]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:36.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:36 smithi196 ceph-mon[76349]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[100622]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[102844]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[102844]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:38.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[100622]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:37 smithi196 ceph-mon[76349]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:39 smithi184 ceph-mon[100622]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:40.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:39 smithi184 ceph-mon[102844]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:40.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:39 smithi196 ceph-mon[76349]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:40.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-mon[100622]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-mon[102844]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:42] "GET /metrics HTTP/1.1" 200 38940 "" "Prometheus/2.43.0" 2024-09-21T03:23:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:42.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:42.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:42.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:42.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:42 smithi196 ceph-mon[76349]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:43.905 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:23:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:44 smithi184 ceph-mon[102844]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:44 smithi184 ceph-mon[100622]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:44.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:44 smithi196 ceph-mon[76349]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 7h - - 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 7s ago 7h - - 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 9s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 7s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 9s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 7s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 9s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 9s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:23:44.448 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 7s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 9s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 9s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 7s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 9s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 9s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 9s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 9s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 7s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 7s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 7s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 7s ago 7h 66.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:23:44.449 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:23:44.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 9s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:23:44.450 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 7s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:23:44.758 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:23:44.758 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:23:44.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:23:44.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:23:44.759 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:23:45.030 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:23:45.031 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:23:45.031 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:23:45.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[100622]: from='client.74709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[100622]: from='client.84677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[100622]: from='client.84683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2773997977' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[102844]: from='client.74709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[102844]: from='client.84677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[102844]: from='client.84683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2773997977' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:45.342 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:23:45.342 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:45.342 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:45.342 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:45.342 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:45 smithi196 ceph-mon[76349]: from='client.74709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:45 smithi196 ceph-mon[76349]: from='client.84677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:45 smithi196 ceph-mon[76349]: from='client.84683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2773997977' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[100622]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[100622]: from='client.84695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1842037354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[102844]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[102844]: from='client.84695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:46 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1842037354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:46.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:46 smithi196 ceph-mon[76349]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:46 smithi196 ceph-mon[76349]: from='client.84695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:46 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1842037354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:48 smithi184 ceph-mon[100622]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:48 smithi184 ceph-mon[102844]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:48 smithi196 ceph-mon[76349]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:50 smithi184 ceph-mon[100622]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:50 smithi184 ceph-mon[102844]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:50 smithi196 ceph-mon[76349]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:52.147 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-mon[100622]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:52.147 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:23:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:23:52.148 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-mon[102844]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:52 smithi196 ceph-mon[76349]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:52.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:52.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:23:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:23:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:23:52.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:23:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:54 smithi184 ceph-mon[100622]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:54 smithi184 ceph-mon[102844]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:54.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:54 smithi196 ceph-mon[76349]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:56.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:56 smithi196 ceph-mon[76349]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:56.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.637 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[102844]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[100622]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:58 smithi196 ceph-mon[76349]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:23:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[100622]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[102844]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:23:58.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:23:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:00 smithi196 ceph-mon[76349]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[100622]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[102844]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-mon[100622]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:02.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-mon[102844]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:02] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:24:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:02.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:02.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:02.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:02.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:02 smithi196 ceph-mon[76349]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:04 smithi196 ceph-mon[76349]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:04 smithi184 ceph-mon[100622]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:04 smithi184 ceph-mon[102844]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:06 smithi196 ceph-mon[76349]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:06 smithi184 ceph-mon[100622]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:06 smithi184 ceph-mon[102844]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:08.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:08 smithi196 ceph-mon[76349]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:24:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:08 smithi184 ceph-mon[100622]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:24:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:08 smithi184 ceph-mon[102844]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:24:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:10 smithi184 ceph-mon[100622]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:24:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:10 smithi184 ceph-mon[102844]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:24:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:10 smithi196 ceph-mon[76349]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:24:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:12.141 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:12] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:24:12.531 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-mon[100622]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:12.531 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-mon[102844]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:12.531 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:12.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:12.531 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:12.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:12.532 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:12.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:12 smithi196 ceph-mon[76349]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:14 smithi184 ceph-mon[100622]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:14 smithi184 ceph-mon[102844]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:14 smithi196 ceph-mon[76349]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:24:15.347 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:15.646 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:24:16.176 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:24:16.176 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 0s ago 7h - - 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 7h - - 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 0s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 19s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 0s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 19s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 0s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 0s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 19s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 0s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 0s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 19s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 0s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 0s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 0s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:24:16.177 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 0s ago 7h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 19s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 19s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 19s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 19s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 19s ago 7h 68.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 0s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:24:16.178 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 19s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:24:16.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:16 smithi196 ceph-mon[76349]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:24:16.380 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.381 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:24:16.486 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:24:16.487 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:24:16.487 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[100622]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[102844]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.488 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:24:16.757 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:24:16.758 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:24:16.758 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:24:16.758 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:24:16.758 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:24:16.758 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:24:17.067 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:24:17.067 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:17.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:17.067 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:17.067 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[100622]: from='client.74745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[100622]: from='client.74751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[100622]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1366799474' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4100133406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[102844]: from='client.74745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[102844]: from='client.74751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[102844]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1366799474' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:17.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:17 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4100133406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:17.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:17 smithi196 ceph-mon[76349]: from='client.74745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:17 smithi196 ceph-mon[76349]: from='client.74751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:17 smithi196 ceph-mon[76349]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:17.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1366799474' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:17.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:17 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4100133406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[100622]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:24:18.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[100622]: from='client.84731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[102844]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[102844]: from='client.84731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:18 smithi196 ceph-mon[76349]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:24:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:18 smithi196 ceph-mon[76349]: from='client.84731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:20 smithi184 ceph-mon[100622]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:24:20.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:20.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:20 smithi184 ceph-mon[102844]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:24:20.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:20 smithi196 ceph-mon[76349]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:24:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-mon[100622]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T03:24:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-mon[102844]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T03:24:22.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:22] "GET /metrics HTTP/1.1" 200 38950 "" "Prometheus/2.43.0" 2024-09-21T03:24:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:22.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:22.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:22.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:22 smithi196 ceph-mon[76349]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T03:24:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:24 smithi184 ceph-mon[100622]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T03:24:24.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:24 smithi184 ceph-mon[102844]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T03:24:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:24 smithi196 ceph-mon[76349]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-21T03:24:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:26 smithi184 ceph-mon[100622]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:26 smithi184 ceph-mon[102844]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:26.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:26 smithi196 ceph-mon[76349]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:28 smithi184 ceph-mon[100622]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:28 smithi184 ceph-mon[102844]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:28 smithi196 ceph-mon[76349]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 431 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:24:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:30.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:30 smithi184 ceph-mon[100622]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:24:30.689 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:30.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:30 smithi184 ceph-mon[102844]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:24:30.689 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:30 smithi196 ceph-mon[76349]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:24:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:32.291 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:24:32.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:32.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:32.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:32.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:32.291 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:32.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:32.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-mon[100622]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:32.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:32 smithi184 ceph-mon[102844]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:32 smithi196 ceph-mon[76349]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:34.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:34 smithi196 ceph-mon[76349]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:34 smithi184 ceph-mon[100622]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:34 smithi184 ceph-mon[102844]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 439 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:24:36.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:36 smithi196 ceph-mon[76349]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T03:24:36.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:36 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[100622]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[102844]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:36.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:36 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:38 smithi196 ceph-mon[76349]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:24:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[100622]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:24:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[102844]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T03:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:40.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:40 smithi196 ceph-mon[76349]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:24:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:40 smithi184 ceph-mon[100622]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:24:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:40 smithi184 ceph-mon[102844]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:24:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:24:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:42.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:42.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:42.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:42 smithi196 ceph-mon[76349]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-mon[100622]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:42 smithi184 ceph-mon[102844]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:44 smithi196 ceph-mon[76349]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:44 smithi184 ceph-mon[100622]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:44 smithi184 ceph-mon[102844]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T03:24:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:46.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:46 smithi196 ceph-mon[76349]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:24:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:46 smithi184 ceph-mon[100622]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:24:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:46 smithi184 ceph-mon[102844]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:24:47.361 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 7h - - 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 10s ago 7h - - 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 12s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 10s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 12s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 10s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 12s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 12s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 10s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 12s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 12s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:24:47.889 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 10s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 12s ago 7h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 12s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 12s ago 7h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 12s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 10s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 10s ago 7h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 10s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 10s ago 7h 69.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 12s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:24:47.890 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 10s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:24:48.204 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:24:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:24:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:24:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:24:48.205 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[100622]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[100622]: from='client.84743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3702928408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[102844]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[102844]: from='client.84743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:48.478 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3702928408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:48.787 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:24:48.787 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:48.787 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:48.787 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:48.788 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:48 smithi196 ceph-mon[76349]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:24:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:48 smithi196 ceph-mon[76349]: from='client.84743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3702928408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[100622]: from='client.74787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[100622]: from='client.84755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[100622]: from='client.84767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3177220427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[102844]: from='client.74787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[102844]: from='client.84755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[102844]: from='client.84767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:49 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3177220427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:49 smithi196 ceph-mon[76349]: from='client.74787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:49 smithi196 ceph-mon[76349]: from='client.84755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:49 smithi196 ceph-mon[76349]: from='client.84767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:49.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:49 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3177220427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:50 smithi184 ceph-mon[100622]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:24:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:50 smithi184 ceph-mon[102844]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:24:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:50.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:50 smithi196 ceph-mon[76349]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:24:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:24:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:24:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:52.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:52.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:24:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:24:52.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:24:52.734 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-mon[100622]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:24:52.734 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:52 smithi184 ceph-mon[102844]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:24:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:52 smithi196 ceph-mon[76349]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T03:24:54.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:54 smithi184 ceph-mon[102844]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:24:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:54 smithi184 ceph-mon[100622]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:24:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:54 smithi196 ceph-mon[76349]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:24:56.690 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:56 smithi196 ceph-mon[76349]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T03:24:56.690 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.690 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.690 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:56 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[100622]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[102844]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:56 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:58 smithi196 ceph-mon[76349]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[100622]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[102844]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:24:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:24:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:24:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:24:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:00 smithi184 ceph-mon[100622]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:00 smithi184 ceph-mon[102844]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:00 smithi196 ceph-mon[76349]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:01.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:01 smithi184 ceph-mon[100622]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:01.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:01 smithi184 ceph-mon[102844]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:01 smithi196 ceph-mon[76349]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:25:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:02.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:02.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:02.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:02.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:04.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:03 smithi184 ceph-mon[100622]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:04.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:03 smithi184 ceph-mon[102844]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:04.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:03 smithi196 ceph-mon[76349]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:06.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:05 smithi184 ceph-mon[100622]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:06.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:05 smithi184 ceph-mon[102844]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:06.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:05 smithi196 ceph-mon[76349]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:08.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:07 smithi184 ceph-mon[100622]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:08.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:07 smithi184 ceph-mon[102844]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:08.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:07 smithi196 ceph-mon[76349]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:09 smithi184 ceph-mon[100622]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:09 smithi184 ceph-mon[102844]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:09 smithi196 ceph-mon[76349]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:12.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:11 smithi184 ceph-mon[100622]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:12.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:11 smithi184 ceph-mon[102844]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:12.048 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:25:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:11 smithi196 ceph-mon[76349]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:25:12.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:12.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:12.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:12.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:12.345 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:12.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:13.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:13 smithi184 ceph-mon[100622]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:13.922 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:13 smithi184 ceph-mon[102844]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:14.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:13 smithi196 ceph-mon[76349]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:15.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:15 smithi196 ceph-mon[76349]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:16.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:15 smithi184 ceph-mon[100622]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:15 smithi184 ceph-mon[102844]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:17.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:16 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:16 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:17 smithi196 ceph-mon[76349]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:18.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[100622]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[102844]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:18.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:19.082 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:25:19.608 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 3s ago 7h - - 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 1s ago 7h - - 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 3s ago 7h 28.3M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 1s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 3s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 1s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 3s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 3s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 1s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 3s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 3s ago 7h 17.6M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:25:19.609 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 1s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 3s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 3s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 3s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 3s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 1s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 1s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 1s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 1s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 1s ago 7h 67.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 3s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:25:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 3s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:25:19.611 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 1s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:25:19.916 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:25:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:25:19.918 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:25:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:19 smithi196 ceph-mon[76349]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:19 smithi196 ceph-mon[76349]: from='client.74817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:19 smithi196 ceph-mon[76349]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:19 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1031062146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:25:20.188 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:25:20.189 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:25:20.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[100622]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:20.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:20.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[100622]: from='client.74817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.189 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[100622]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1031062146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[102844]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[102844]: from='client.74817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[102844]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:20.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:19 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1031062146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:20.496 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:25:20.496 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:20.496 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:20.497 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:20.497 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[100622]: from='client.84791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[100622]: from='client.74841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3765426177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[102844]: from='client.84791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[102844]: from='client.74841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3765426177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:20 smithi196 ceph-mon[76349]: from='client.84791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:20 smithi196 ceph-mon[76349]: from='client.74841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3765426177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:22.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:21 smithi184 ceph-mon[100622]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:25:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:21 smithi184 ceph-mon[102844]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:22.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:22.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:22.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:22.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:21 smithi196 ceph-mon[76349]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:23 smithi184 ceph-mon[100622]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:23 smithi184 ceph-mon[102844]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:23 smithi196 ceph-mon[76349]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:25 smithi184 ceph-mon[100622]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:25 smithi184 ceph-mon[102844]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:25 smithi196 ceph-mon[76349]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:27 smithi184 ceph-mon[100622]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:27 smithi184 ceph-mon[102844]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:27 smithi196 ceph-mon[76349]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[100622]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[102844]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:30.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:29 smithi196 ceph-mon[76349]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:31 smithi184 ceph-mon[100622]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:31 smithi184 ceph-mon[102844]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:32] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:25:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:32.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:32.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:32.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:32.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:32.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:31 smithi196 ceph-mon[76349]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:34.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:33 smithi184 ceph-mon[100622]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:34.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:33 smithi184 ceph-mon[102844]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:34.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:33 smithi196 ceph-mon[76349]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:36.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:35 smithi184 ceph-mon[100622]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:35 smithi184 ceph-mon[102844]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:35 smithi196 ceph-mon[76349]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:38 smithi196 ceph-mon[76349]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[100622]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[102844]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:40 smithi196 ceph-mon[76349]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:40 smithi184 ceph-mon[100622]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:40 smithi184 ceph-mon[102844]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:42.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-mon[100622]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-mon[102844]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:42.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:42] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:25:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:42.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:42.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:42.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:42 smithi196 ceph-mon[76349]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:44 smithi196 ceph-mon[76349]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:44.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:44 smithi184 ceph-mon[100622]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:44 smithi184 ceph-mon[102844]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:45.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:46 smithi196 ceph-mon[76349]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:46 smithi184 ceph-mon[100622]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:46.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:46 smithi184 ceph-mon[102844]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:48 smithi196 ceph-mon[76349]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:48 smithi184 ceph-mon[100622]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:48.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:48 smithi184 ceph-mon[102844]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:50 smithi196 ceph-mon[76349]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:50 smithi184 ceph-mon[100622]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:50 smithi184 ceph-mon[102844]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:50.792 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:25:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:25:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 7h - - 2024-09-21T03:25:51.317 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 13s ago 7h - - 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 15s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 13s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 15s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 13s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 15s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 15s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 13s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 15s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 15s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 13s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 15s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 15s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 15s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:25:51.318 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 15s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 13s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 13s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 13s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 13s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 13s ago 7h 67.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 15s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 15s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:25:51.319 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 13s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:51.626 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:25:51.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:25:51.894 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:25:51.894 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:25:51.894 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:25:51.894 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:25:51.894 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:25:51.895 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:25:52.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[100622]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:52.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[100622]: from='client.74853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[100622]: from='client.74859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[100622]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.153 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3858789088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:25:52] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[102844]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[102844]: from='client.74853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[102844]: from='client.74859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[102844]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.154 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3858789088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:52.210 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:25:52.210 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:52.210 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:52.210 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:52.210 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:52 smithi196 ceph-mon[76349]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:52 smithi196 ceph-mon[76349]: from='client.74853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:52 smithi196 ceph-mon[76349]: from='client.74859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:52 smithi196 ceph-mon[76349]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3858789088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:52.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:52.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:25:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:25:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:25:52.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:25:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:53 smithi196 ceph-mon[76349]: from='client.84839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:53.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:53 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2452630077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:53.458 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:53 smithi184 ceph-mon[100622]: from='client.84839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:53.459 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:53 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2452630077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:53.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:53 smithi184 ceph-mon[102844]: from='client.84839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:53.459 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:53 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2452630077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:54 smithi196 ceph-mon[76349]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:54.534 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:54 smithi184 ceph-mon[100622]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:54.534 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:54 smithi184 ceph-mon[102844]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:56 smithi196 ceph-mon[76349]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:56.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:56 smithi184 ceph-mon[100622]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:56 smithi184 ceph-mon[102844]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:57.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:58 smithi196 ceph-mon[76349]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:58.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:25:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[102844]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[100622]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:25:58.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:25:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:00.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:00 smithi196 ceph-mon[76349]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:00.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:00.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[100622]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[102844]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:00.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-mon[100622]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-mon[102844]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:26:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:02.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:02.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:02.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:02.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:02 smithi196 ceph-mon[76349]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:04 smithi184 ceph-mon[100622]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:04 smithi184 ceph-mon[102844]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:04 smithi196 ceph-mon[76349]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:06 smithi184 ceph-mon[100622]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:06.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:06 smithi184 ceph-mon[102844]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:06 smithi196 ceph-mon[76349]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:08 smithi184 ceph-mon[100622]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:08.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:08 smithi184 ceph-mon[102844]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:08 smithi196 ceph-mon[76349]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:10 smithi184 ceph-mon[100622]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:10 smithi184 ceph-mon[102844]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:10.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:10 smithi196 ceph-mon[76349]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-mon[100622]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:12.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-mon[102844]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:26:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:12.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:12.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:12.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:12 smithi196 ceph-mon[76349]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:14.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:14 smithi196 ceph-mon[76349]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:14.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:14 smithi184 ceph-mon[100622]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:14.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:14 smithi184 ceph-mon[102844]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:14.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:16.503 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:16 smithi184 ceph-mon[100622]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:16.503 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:16 smithi184 ceph-mon[102844]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:16 smithi196 ceph-mon[76349]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:17.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:18 smithi196 ceph-mon[76349]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[100622]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[102844]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:20 smithi196 ceph-mon[76349]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:20 smithi184 ceph-mon[102844]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:20.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:20 smithi184 ceph-mon[100622]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:20.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:26:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:22.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:22.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:22.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:22.513 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:26:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:22 smithi196 ceph-mon[76349]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-mon[100622]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:22 smithi184 ceph-mon[102844]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 6s ago 7h - - 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 6s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 5s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 6s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 5s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 6s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:26:23.043 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 6s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 5s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 6s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 6s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 5s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 6s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 6s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 6s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 6s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 5s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 5s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 5s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 5s ago 7h 69.9M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:26:23.044 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:26:23.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:26:23.045 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 5s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:26:23.353 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:26:23.353 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:26:23.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:26:23.355 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:26:23.623 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:26:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:26:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:26:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:26:23.623 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:26:23.624 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:26:23.624 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:23 smithi184 ceph-mon[100622]: from='client.74889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:23.625 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:23 smithi184 ceph-mon[102844]: from='client.74889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:23.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:23 smithi196 ceph-mon[76349]: from='client.74889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:23.932 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:26:23.932 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:23.933 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:23.933 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:23.933 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:24.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:24 smithi196 ceph-mon[76349]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:24 smithi196 ceph-mon[76349]: from='client.84857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:24 smithi196 ceph-mon[76349]: from='client.84863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/638689325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:24 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4151147065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[102844]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[102844]: from='client.84857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[102844]: from='client.84863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/638689325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4151147065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[100622]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[100622]: from='client.84857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[100622]: from='client.84863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/638689325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:24 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4151147065' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:25.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:25 smithi196 ceph-mon[76349]: from='client.74913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:25.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:25 smithi184 ceph-mon[100622]: from='client.74913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:25.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:25 smithi184 ceph-mon[102844]: from='client.74913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:26 smithi196 ceph-mon[76349]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:26 smithi184 ceph-mon[100622]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:26 smithi184 ceph-mon[102844]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:28 smithi196 ceph-mon[76349]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:28 smithi184 ceph-mon[100622]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:28 smithi184 ceph-mon[102844]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:30 smithi196 ceph-mon[76349]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:30 smithi184 ceph-mon[102844]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:30 smithi184 ceph-mon[100622]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:32] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:26:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:32.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:32.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:32.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-mon[100622]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:32 smithi184 ceph-mon[102844]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:32 smithi196 ceph-mon[76349]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:34.712 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:34 smithi196 ceph-mon[76349]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:34.775 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:34 smithi184 ceph-mon[100622]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:34.776 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:34 smithi184 ceph-mon[102844]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:36 smithi184 ceph-mon[100622]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:36 smithi184 ceph-mon[102844]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:36 smithi196 ceph-mon[76349]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[100622]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[102844]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:38 smithi196 ceph-mon[76349]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:38.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:40 smithi184 ceph-mon[100622]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:40 smithi184 ceph-mon[102844]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:40.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:40 smithi196 ceph-mon[76349]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:40.874 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:42] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:26:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:42.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:42.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:42.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-mon[102844]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:42 smithi184 ceph-mon[100622]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:42 smithi196 ceph-mon[76349]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:44 smithi184 ceph-mon[102844]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:44 smithi184 ceph-mon[100622]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:44 smithi196 ceph-mon[76349]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:46 smithi184 ceph-mon[102844]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:46 smithi184 ceph-mon[100622]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:46 smithi196 ceph-mon[76349]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:26:48.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:48 smithi184 ceph-mon[102844]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:48.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:48 smithi184 ceph-mon[100622]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:48 smithi196 ceph-mon[76349]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:50 smithi184 ceph-mon[100622]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:50 smithi184 ceph-mon[102844]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:50 smithi196 ceph-mon[76349]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:26:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:26:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:52.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:52.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:26:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:26:52.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:26:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-mon[100622]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:52 smithi184 ceph-mon[102844]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:52 smithi196 ceph-mon[76349]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:54.226 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:26:54.754 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:26:54.754 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 18s ago 7h - - 2024-09-21T03:26:54.754 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 16s ago 7h - - 2024-09-21T03:26:54.754 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 18s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 16s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 18s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 16s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 18s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 18s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 16s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 18s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:26:54.755 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 18s ago 7h 17.5M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 16s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 18s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 18s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 18s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 18s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 16s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 16s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 16s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:26:54.756 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 16s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:26:54.757 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 16s ago 7h 67.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:26:54.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 18s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:26:54.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:26:54.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 18s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:26:54.757 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 16s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:26:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:54 smithi184 ceph-mon[100622]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:54.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:54 smithi184 ceph-mon[102844]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:54 smithi196 ceph-mon[76349]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:26:55.064 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:26:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:26:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:26:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:26:55.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:26:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:26:55.339 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:26:55.339 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:26:55.339 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:26:55.339 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:26:55.340 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:26:55.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[100622]: from='client.74925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[100622]: from='client.84893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1526243459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:55.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[102844]: from='client.74925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[102844]: from='client.84893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:55 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1526243459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:55.664 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:26:55.664 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:55.664 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:55.664 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:55.664 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:55 smithi196 ceph-mon[76349]: from='client.74925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:55 smithi196 ceph-mon[76349]: from='client.84893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:55.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:55 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1526243459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:56.631 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[100622]: from='client.84899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[100622]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[100622]: from='client.84911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/833591612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[102844]: from='client.84899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[102844]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[102844]: from='client.84911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:56 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/833591612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:56 smithi196 ceph-mon[76349]: from='client.84899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:56 smithi196 ceph-mon[76349]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:26:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:56 smithi196 ceph-mon[76349]: from='client.84911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:56.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:56 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/833591612' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:57.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[100622]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:57 smithi184 ceph-mon[102844]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:57.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:57 smithi196 ceph-mon[76349]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:26:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:26:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:26:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:26:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:00 smithi196 ceph-mon[76349]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T03:27:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:00 smithi184 ceph-mon[100622]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T03:27:00.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:00 smithi184 ceph-mon[102844]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-09-21T03:27:00.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:02.294 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:27:02.295 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:02.157Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:02.295 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:02.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:02.295 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:02.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:02 smithi196 ceph-mon[76349]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-mon[100622]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:02.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:02 smithi184 ceph-mon[102844]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:04 smithi184 ceph-mon[100622]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:04 smithi184 ceph-mon[102844]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:04.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:04 smithi196 ceph-mon[76349]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:27:06.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:06 smithi196 ceph-mon[76349]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:27:06.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:06 smithi184 ceph-mon[100622]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:27:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:06 smithi184 ceph-mon[102844]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:27:08.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:08 smithi196 ceph-mon[76349]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:27:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:08 smithi184 ceph-mon[100622]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:27:08.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:08 smithi184 ceph-mon[102844]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:10 smithi196 ceph-mon[76349]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:27:10.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:10 smithi184 ceph-mon[100622]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:27:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:10 smithi184 ceph-mon[102844]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:27:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:27:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:12.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:12.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:12.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:12.639 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:12 smithi196 ceph-mon[76349]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-mon[100622]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:12 smithi184 ceph-mon[102844]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:14 smithi196 ceph-mon[76349]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:14.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:14 smithi184 ceph-mon[100622]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:14 smithi184 ceph-mon[102844]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:16 smithi184 ceph-mon[100622]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:16 smithi184 ceph-mon[102844]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:16.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:16 smithi196 ceph-mon[76349]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[100622]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:17 smithi184 ceph-mon[102844]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:17 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:18.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:17 smithi196 ceph-mon[76349]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:20 smithi184 ceph-mon[100622]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:20.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:20 smithi184 ceph-mon[102844]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:20.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:20 smithi196 ceph-mon[76349]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:22.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:22.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:22.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:22 smithi196 ceph-mon[76349]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:22.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-mon[100622]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:22 smithi184 ceph-mon[102844]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:24 smithi196 ceph-mon[76349]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:24.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:24 smithi184 ceph-mon[100622]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:24.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:24 smithi184 ceph-mon[102844]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:25.960 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 9s ago 7h - - 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 7h - - 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 9s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 8s ago 7h 96.1M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:27:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 9s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 8s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 9s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 9s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 8s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 9s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 9s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 8s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 9s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 9s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 9s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 9s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 8s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:27:26.491 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 8s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 8s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 8s ago 7h 68.3M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 9s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 9s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:27:26.492 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 8s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:27:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:26 smithi184 ceph-mon[100622]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:26 smithi184 ceph-mon[102844]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:26.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:27:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:27:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:27:26.801 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:27:26.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:26 smithi196 ceph-mon[76349]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:27.068 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:27:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:27:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:27:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:27:27.069 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:27:27.381 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:27:27.381 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:27.381 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:27.381 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:27.382 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[102844]: from='client.74961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[102844]: from='client.74967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[102844]: from='client.74973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4004658938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2415077289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[100622]: from='client.74961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[100622]: from='client.74967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[100622]: from='client.74973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4004658938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:27.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:27 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2415077289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:27.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:27 smithi196 ceph-mon[76349]: from='client.74961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:27 smithi196 ceph-mon[76349]: from='client.74967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:27 smithi196 ceph-mon[76349]: from='client.74973 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4004658938' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:27.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:27 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2415077289' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:28 smithi184 ceph-mon[100622]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:28.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:28 smithi184 ceph-mon[100622]: from='client.84947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:28 smithi184 ceph-mon[102844]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:28.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:28 smithi184 ceph-mon[102844]: from='client.84947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:28 smithi196 ceph-mon[76349]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:28.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:28 smithi196 ceph-mon[76349]: from='client.84947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:29.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:29.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:29.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:30 smithi184 ceph-mon[100622]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:30 smithi184 ceph-mon[102844]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:30 smithi196 ceph-mon[76349]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:27:32.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:32.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:32.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:32.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:32.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-mon[100622]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:32.758 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:32 smithi184 ceph-mon[102844]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:32 smithi196 ceph-mon[76349]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:34 smithi184 ceph-mon[102844]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:34 smithi184 ceph-mon[100622]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:34 smithi196 ceph-mon[76349]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:36.700 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:36 smithi184 ceph-mon[102844]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:36.700 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:36 smithi184 ceph-mon[100622]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:36 smithi196 ceph-mon[76349]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:38.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[100622]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:38.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.030 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[102844]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:38.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.031 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:37 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:37 smithi196 ceph-mon[76349]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:38.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:37 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:39.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:40 smithi184 ceph-mon[100622]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:40 smithi184 ceph-mon[102844]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:40 smithi196 ceph-mon[76349]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:41.736 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:41 smithi196 ceph-mon[76349]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:42.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:41 smithi184 ceph-mon[100622]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:42.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:41 smithi184 ceph-mon[102844]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:27:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:42.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:42.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:42.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:43.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:43 smithi196 ceph-mon[76349]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:43 smithi184 ceph-mon[100622]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:43 smithi184 ceph-mon[102844]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:45.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:45 smithi196 ceph-mon[76349]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:45 smithi184 ceph-mon[100622]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:45 smithi184 ceph-mon[102844]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:48.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:47 smithi196 ceph-mon[76349]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:47 smithi184 ceph-mon[100622]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:47 smithi184 ceph-mon[102844]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:49 smithi196 ceph-mon[76349]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:50.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:49 smithi184 ceph-mon[100622]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:49 smithi184 ceph-mon[102844]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:52.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:51 smithi196 ceph-mon[76349]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.161 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:51 smithi184 ceph-mon[100622]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.162 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:27:52] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:27:52.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:51 smithi184 ceph-mon[102844]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:52.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:52.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:27:52.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:27:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:27:52.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:27:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:53 smithi196 ceph-mon[76349]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:53 smithi184 ceph-mon[100622]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:54.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:53 smithi184 ceph-mon[102844]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:56.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:55 smithi196 ceph-mon[76349]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:55 smithi184 ceph-mon[100622]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:56.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:55 smithi184 ceph-mon[102844]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:57.675 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 1s ago 7h - - 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 19s ago 7h - - 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 1s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 19s ago 7h 93.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 1s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 19s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 1s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:27:58.211 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 1s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 19s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 1s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 1s ago 7h 17.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 19s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 1s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 1s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 1s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 19s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 19s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 19s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:27:58.212 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 19s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:27:58.213 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 19s ago 7h 71.0M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:27:58.213 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 1s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:27:58.213 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:27:58.213 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 1s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:27:58.213 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 19s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:27:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[100622]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[102844]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:57 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:57 smithi196 ceph-mon[76349]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:57 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:58.552 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:58.579 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:27:58.580 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:27:58.832 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:27:58.832 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:27:58.832 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:27:58.832 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:27:58.832 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:27:58.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:27:59.206 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:27:59.206 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:59.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:59.207 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:59.207 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='client.74997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='client.75009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/212915946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='client.74997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='client.75009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/212915946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:59.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='client.74997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='client.75009 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/212915946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:27:59.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[100622]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[100622]: from='client.75021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/728466282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[102844]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[102844]: from='client.75021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/728466282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:27:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:59 smithi196 ceph-mon[76349]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:59 smithi196 ceph-mon[76349]: from='client.75021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:59 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/728466282' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:27:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:02.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:01 smithi184 ceph-mon[100622]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:01 smithi184 ceph-mon[102844]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:02.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:02] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:28:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:02.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:02.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:02.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:02.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:01 smithi196 ceph-mon[76349]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:04.270 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:03 smithi184 ceph-mon[100622]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:04.270 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:03 smithi184 ceph-mon[102844]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:04.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:03 smithi196 ceph-mon[76349]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:06.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:05 smithi184 ceph-mon[100622]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:06.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:05 smithi184 ceph-mon[102844]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:05 smithi196 ceph-mon[76349]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:08.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:07 smithi184 ceph-mon[100622]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:08.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:07 smithi184 ceph-mon[102844]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:07 smithi196 ceph-mon[76349]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:09 smithi184 ceph-mon[100622]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:10.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:09 smithi184 ceph-mon[102844]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:10.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:09 smithi196 ceph-mon[76349]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:11 smithi184 ceph-mon[100622]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:12] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:28:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:11 smithi184 ceph-mon[102844]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:12.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:12.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:12.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:12.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:11 smithi196 ceph-mon[76349]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:13 smithi184 ceph-mon[100622]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:14.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:13 smithi184 ceph-mon[102844]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:14.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:13 smithi196 ceph-mon[76349]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:14.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Sep 21 03:28:13 smithi196 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-grafana-a[46776]: logger=cleanup t=2024-09-21T03:28:13.8107002Z level=info msg="Completed cleanup jobs" duration=11.204457ms 2024-09-21T03:28:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:16 smithi184 ceph-mon[100622]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:16 smithi184 ceph-mon[102844]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:16 smithi196 ceph-mon[76349]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:18.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[100622]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:18.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.272 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.272 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[102844]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:18.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:18 smithi196 ceph-mon[76349]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.912 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[100622]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[102844]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:19.913 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:19 smithi196 ceph-mon[76349]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:20.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:22.126 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:21 smithi184 ceph-mon[100622]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:22.127 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:21 smithi184 ceph-mon[102844]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:22.127 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:28:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:21 smithi196 ceph-mon[76349]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:22.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:22.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:22.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:22.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:24.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:23 smithi196 ceph-mon[76349]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:23 smithi184 ceph-mon[100622]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:23 smithi184 ceph-mon[102844]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:25 smithi196 ceph-mon[76349]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:25 smithi184 ceph-mon[100622]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:26.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:25 smithi184 ceph-mon[102844]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:28.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:27 smithi196 ceph-mon[76349]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:27 smithi184 ceph-mon[100622]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:28.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:27 smithi184 ceph-mon[102844]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:28:30.039 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 12s ago 7h - - 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 12s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 11s ago 7h 95.7M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 12s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 11s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 12s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 12s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 11s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 12s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 12s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:28:30.040 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 11s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 12s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 12s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 12s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 12s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 11s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 11s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 11s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 11s ago 7h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 12s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 12s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:28:30.041 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 11s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:29 smithi196 ceph-mon[76349]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:30.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:29 smithi196 ceph-mon[76349]: from='client.84995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[100622]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[100622]: from='client.84995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[102844]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:29 smithi184 ceph-mon[102844]: from='client.84995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.352 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:28:30.352 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:28:30.352 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:28:30.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:28:30.353 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:28:30.622 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:28:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:28:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:28:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:28:30.623 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:28:30.932 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:28:30.932 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:30.932 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:30.932 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:30.932 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[100622]: from='client.85001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[100622]: from='client.85007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/935520320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[102844]: from='client.85001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[102844]: from='client.85007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:30.933 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:30 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/935520320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:30 smithi196 ceph-mon[76349]: from='client.85001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:30 smithi196 ceph-mon[76349]: from='client.85007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:31.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:30 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/935520320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[100622]: from='client.85019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[100622]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:32.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1556284749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[102844]: from='client.85019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[102844]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:32.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:31 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1556284749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:31 smithi196 ceph-mon[76349]: from='client.85019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:31 smithi196 ceph-mon[76349]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:32.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:31 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1556284749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:28:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:32.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:32.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:32.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:34.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:33 smithi196 ceph-mon[76349]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:33 smithi184 ceph-mon[100622]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:34.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:33 smithi184 ceph-mon[102844]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:36.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:35 smithi196 ceph-mon[76349]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:35 smithi184 ceph-mon[100622]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:35 smithi184 ceph-mon[102844]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[102844]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[100622]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:38 smithi196 ceph-mon[76349]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:38.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[102844]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[100622]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:39 smithi196 ceph-mon[76349]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:40.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:42.265 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:41 smithi184 ceph-mon[100622]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:42.266 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:41 smithi184 ceph-mon[102844]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:42.266 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:28:42.266 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:42.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:42.267 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:42.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:42.267 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:42.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:42.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:41 smithi196 ceph-mon[76349]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:44.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:43 smithi196 ceph-mon[76349]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:43 smithi184 ceph-mon[100622]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:44.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:43 smithi184 ceph-mon[102844]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:45.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:46.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:45 smithi196 ceph-mon[76349]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:45 smithi184 ceph-mon[100622]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:46.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:45 smithi184 ceph-mon[102844]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:48.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:47 smithi184 ceph-mon[100622]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:48.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:47 smithi184 ceph-mon[102844]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:47 smithi196 ceph-mon[76349]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:50.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:49 smithi184 ceph-mon[100622]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:50.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:49 smithi184 ceph-mon[102844]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:50.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:49 smithi196 ceph-mon[76349]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:52.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:51 smithi184 ceph-mon[100622]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:52.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:51 smithi184 ceph-mon[102844]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:28:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:28:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:52.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:52.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:28:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:28:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:28:52.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:28:52.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:51 smithi196 ceph-mon[76349]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:53 smithi184 ceph-mon[100622]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:53 smithi184 ceph-mon[102844]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:54.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:53 smithi196 ceph-mon[76349]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:55 smithi184 ceph-mon[100622]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:55 smithi184 ceph-mon[102844]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:55 smithi196 ceph-mon[76349]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:58 smithi196 ceph-mon[76349]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:28:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[102844]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[100622]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:28:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:28:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:00.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:00.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:01.239 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:29:01.822 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:29:01.822 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 4s ago 7h - - 2024-09-21T03:29:01.822 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 2s ago 7h - - 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 4s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 2s ago 7h 94.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 4s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 2s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 4s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 4s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 4s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 4s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 2s ago 7h 17.9M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 4s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 4s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 4s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:29:01.823 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 4s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 2s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 2s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 2s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 2s ago 7h 66.7M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 4s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 4s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:29:01.824 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 2s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:29:02.146 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:29:02.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:02.148 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:29:02.148 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:29:02.148 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:29:02.148 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:29:02.148 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:29:02.148 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[100622]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:02.148 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[100622]: from='client.75069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.148 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[100622]: from='client.85037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[102844]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:02.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[102844]: from='client.75069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.149 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-mon[102844]: from='client.85037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:02 smithi196 ceph-mon[76349]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:02 smithi196 ceph-mon[76349]: from='client.75069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:02 smithi196 ceph-mon[76349]: from='client.85037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:29:02.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:29:02.429 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:29:02.429 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:02.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:02.429 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:02.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:02.429 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:02.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:02.739 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:29:02.740 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:02.740 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:03 smithi196 ceph-mon[76349]: from='client.85043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1131093748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:03 smithi196 ceph-mon[76349]: from='client.85055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:03 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2125153879' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[100622]: from='client.85043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1131093748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[100622]: from='client.85055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2125153879' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[102844]: from='client.85043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1131093748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[102844]: from='client.85055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:03.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:03 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2125153879' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:04 smithi196 ceph-mon[76349]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:04 smithi184 ceph-mon[100622]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:04.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:04 smithi184 ceph-mon[102844]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:06 smithi196 ceph-mon[76349]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:06.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:06 smithi184 ceph-mon[100622]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:06.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:06 smithi184 ceph-mon[102844]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:08 smithi196 ceph-mon[76349]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:08.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:08 smithi184 ceph-mon[100622]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:08.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:08 smithi184 ceph-mon[102844]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:10 smithi196 ceph-mon[76349]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:10 smithi184 ceph-mon[100622]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:10.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:10 smithi184 ceph-mon[102844]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:10.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-mon[100622]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-mon[102844]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:29:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:12.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:12.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:12.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:12 smithi196 ceph-mon[76349]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:14.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:14 smithi196 ceph-mon[76349]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:14.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:14 smithi184 ceph-mon[100622]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:14.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:14 smithi184 ceph-mon[102844]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:15 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:15.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:15 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:15.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:15 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:16 smithi196 ceph-mon[76349]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:16.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:16 smithi184 ceph-mon[100622]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:16.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:16 smithi184 ceph-mon[102844]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[100622]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[102844]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:18 smithi196 ceph-mon[76349]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[100622]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[102844]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:20 smithi196 ceph-mon[76349]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:20.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:22.167 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:22] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:29:22.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-mon[100622]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:22.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-mon[102844]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:22.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:22.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:22.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:22.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:22.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:22.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:22 smithi196 ceph-mon[76349]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:24 smithi184 ceph-mon[100622]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:24 smithi184 ceph-mon[102844]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:24.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:24 smithi196 ceph-mon[76349]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:26 smithi196 ceph-mon[76349]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:26 smithi184 ceph-mon[100622]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:26.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:26 smithi184 ceph-mon[102844]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:28 smithi184 ceph-mon[100622]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:28.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:28 smithi184 ceph-mon[102844]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:28 smithi196 ceph-mon[76349]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[100622]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:30.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[102844]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:30.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:30 smithi196 ceph-mon[76349]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-mon[100622]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-mon[102844]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:32] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:29:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:32.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:32.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:32.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:32 smithi196 ceph-mon[76349]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:33.038 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 16s ago 7h 28.4M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 96.6M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 16s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 14s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:29:33.562 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 16s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 16s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 14s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 16s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 16s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 14s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 16s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 16s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 16s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 14s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 14s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 14s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 14s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 14s ago 7h 67.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:29:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:29:33.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:29:33.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:29:33.564 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 14s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:29:33.873 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:29:33.873 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:29:33.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:29:33.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:29:33.874 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:29:34.144 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:29:34.144 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:29:34.144 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:29:34.145 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:29:34.455 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:29:34.455 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:34.455 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:34.455 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:34.456 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[102844]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[102844]: from='client.75105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[102844]: from='client.85073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[102844]: from='client.85079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1777051451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[100622]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[100622]: from='client.75105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.456 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[100622]: from='client.85073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[100622]: from='client.85079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.457 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:34 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1777051451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:34 smithi196 ceph-mon[76349]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:34 smithi196 ceph-mon[76349]: from='client.75105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:34 smithi196 ceph-mon[76349]: from='client.85073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:34 smithi196 ceph-mon[76349]: from='client.85079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:34 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1777051451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:35.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:35 smithi196 ceph-mon[76349]: from='client.85091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:35.541 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:35 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2531451035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:35.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:35 smithi184 ceph-mon[100622]: from='client.85091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:35.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:35 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2531451035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:35 smithi184 ceph-mon[102844]: from='client.85091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:35.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:35 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2531451035' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:36.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:36 smithi184 ceph-mon[100622]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:36.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:36 smithi184 ceph-mon[102844]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:36.555 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:36 smithi196 ceph-mon[76349]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[100622]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:38.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[102844]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:38 smithi196 ceph-mon[76349]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:40 smithi196 ceph-mon[76349]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[100622]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[102844]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:40.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:42.213 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:42] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:29:42.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:42.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:42.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:42.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:42.213 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:42.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:42.468 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-mon[100622]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:42.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:42 smithi184 ceph-mon[102844]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:42 smithi196 ceph-mon[76349]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:44 smithi196 ceph-mon[76349]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:44.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:44 smithi184 ceph-mon[100622]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:44 smithi184 ceph-mon[102844]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:46.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:46 smithi196 ceph-mon[76349]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:46 smithi184 ceph-mon[100622]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:46 smithi184 ceph-mon[102844]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:48 smithi196 ceph-mon[76349]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:48 smithi184 ceph-mon[100622]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:48 smithi184 ceph-mon[102844]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:50 smithi196 ceph-mon[76349]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:50 smithi184 ceph-mon[100622]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:50 smithi184 ceph-mon[102844]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:29:52] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:29:52.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:52.168Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:52.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:29:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:29:52.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:29:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:52 smithi196 ceph-mon[76349]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-mon[100622]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:52 smithi184 ceph-mon[102844]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:54.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:54 smithi196 ceph-mon[76349]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:54 smithi184 ceph-mon[100622]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:54 smithi184 ceph-mon[102844]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:56.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:56 smithi196 ceph-mon[76349]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:56 smithi184 ceph-mon[100622]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:56 smithi184 ceph-mon[102844]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:58 smithi196 ceph-mon[76349]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[100622]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[102844]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:29:59.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:29:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:59.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:29:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:59.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:29:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mon-a[100618]: 2024-09-21T03:29:59.998+0000 7f19287c4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[100622]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:00 smithi184 ceph-mon[102844]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:00 smithi196 ceph-mon[76349]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:02.256 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:02] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:30:02.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:02.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:02.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:02.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:02.256 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:02.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:02.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-mon[100622]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:02.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:02 smithi184 ceph-mon[102844]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:02 smithi196 ceph-mon[76349]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:04.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:04 smithi184 ceph-mon[100622]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:04.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:04 smithi184 ceph-mon[102844]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:04.739 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:30:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:04 smithi196 ceph-mon[76349]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:05.273 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 7s ago 7h - - 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 5s ago 7h - - 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 7s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 5s ago 7h 94.9M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 7s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 5s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 7s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 7s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 5s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 7s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 7s ago 7h 17.8M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 5s ago 7h 17.1M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:30:05.274 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 7s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 7s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 7s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 7s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 5s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 5s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 5s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 5s ago 7h 69.8M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 7s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 7s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:30:05.275 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 5s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:30:05.585 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:30:05.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:30:05.585 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:30:05.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:30:05.586 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:30:05.853 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:30:05.853 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:30:05.853 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:30:05.853 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:30:05.853 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:30:05.854 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:30:06.161 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:06.161 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:06.161 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:06.161 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:06.161 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:06.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: from='client.75141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: from='client.85109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: from='client.75153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/655611667' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3596082540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: from='client.75141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: from='client.85109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: from='client.75153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/655611667' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:06 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3596082540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: from='client.75141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: from='client.85109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: from='client.75153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/655611667' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:06 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3596082540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:07.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:07 smithi196 ceph-mon[76349]: from='client.85127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:07.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:07 smithi184 ceph-mon[100622]: from='client.85127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:07.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:07 smithi184 ceph-mon[102844]: from='client.85127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:08.758 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:08 smithi196 ceph-mon[76349]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:08 smithi184 ceph-mon[100622]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:08 smithi184 ceph-mon[102844]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:10 smithi184 ceph-mon[100622]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:10.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:10 smithi184 ceph-mon[102844]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:10 smithi196 ceph-mon[76349]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:12] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:30:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:12.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:12.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:12.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:12.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-mon[100622]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:12.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:12 smithi184 ceph-mon[102844]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:12 smithi196 ceph-mon[76349]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:14.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:14 smithi184 ceph-mon[100622]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:14.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:14 smithi184 ceph-mon[102844]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:14.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:14 smithi196 ceph-mon[76349]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:16.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:16 smithi184 ceph-mon[100622]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:16.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:16 smithi184 ceph-mon[102844]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:16 smithi196 ceph-mon[76349]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:18.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:18 smithi196 ceph-mon[76349]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[100622]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[102844]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:18.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:20 smithi196 ceph-mon[76349]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[100622]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[102844]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:21.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:21.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.050 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:21.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:21 smithi196 ceph-mon[76349]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:21.926 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:21 smithi184 ceph-mon[100622]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:21.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:21 smithi184 ceph-mon[102844]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:22.229 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:22] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:30:22.229 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:22.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:22.229 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:22.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:22.229 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:22.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:23 smithi184 ceph-mon[100622]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:24.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:23 smithi184 ceph-mon[102844]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:23 smithi196 ceph-mon[76349]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:25 smithi184 ceph-mon[100622]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:25 smithi184 ceph-mon[102844]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:25 smithi196 ceph-mon[76349]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:28.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:27 smithi196 ceph-mon[76349]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:27 smithi184 ceph-mon[100622]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:27 smithi184 ceph-mon[102844]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[100622]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[102844]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:29 smithi196 ceph-mon[76349]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:30.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:30:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:32.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:32.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:32.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-mon[102844]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:32 smithi184 ceph-mon[100622]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:32 smithi196 ceph-mon[76349]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:34 smithi184 ceph-mon[100622]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:34.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:34 smithi184 ceph-mon[102844]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:34.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:34 smithi196 ceph-mon[76349]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:36.459 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:30:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:36 smithi184 ceph-mon[100622]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:36 smithi184 ceph-mon[102844]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:36.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:36 smithi196 ceph-mon[76349]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:36.987 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 17s ago 7h - - 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 17s ago 7h 95.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 19s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 17s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 19s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 19s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 17s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:30:36.988 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 19s ago 7h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 18.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 17s ago 7h 17.4M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 19s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 19s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 19s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 17s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 17s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 17s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 17s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 17s ago 7h 70.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:30:36.989 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 17s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:30:37.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:30:37.298 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:30:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:37 smithi184 ceph-mon[100622]: from='client.75177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:37.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:37 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2521773840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:37.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:37 smithi184 ceph-mon[102844]: from='client.75177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:37.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:37 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2521773840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:30:37.565 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:30:37.566 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:30:37.874 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:37.874 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:37.874 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:37.874 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:37.874 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:37 smithi196 ceph-mon[76349]: from='client.75177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:37.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:37 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2521773840' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='client.85145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='client.85151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='client.75201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3462798878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='client.85145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='client.85151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='client.75201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3462798878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.800 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:38 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='client.85145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='client.85151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='client.75201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3462798878' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:38.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:38 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[100622]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:40.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[102844]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:40 smithi196 ceph-mon[76349]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:30:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:42.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:42.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:42.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-mon[102844]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:42 smithi184 ceph-mon[100622]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:42 smithi196 ceph-mon[76349]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:44.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:44 smithi184 ceph-mon[102844]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:44.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:44 smithi184 ceph-mon[100622]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:44.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:44 smithi196 ceph-mon[76349]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:46 smithi184 ceph-mon[100622]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:46.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:46 smithi184 ceph-mon[102844]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:46 smithi196 ceph-mon[76349]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:48 smithi196 ceph-mon[76349]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:48 smithi184 ceph-mon[100622]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:48 smithi184 ceph-mon[102844]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:50 smithi196 ceph-mon[76349]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:50 smithi184 ceph-mon[100622]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:50 smithi184 ceph-mon[102844]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:52.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:30:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:30:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:52.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:52.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:30:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:30:52.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:30:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-mon[100622]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:52 smithi184 ceph-mon[102844]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:52 smithi196 ceph-mon[76349]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:54 smithi184 ceph-mon[100622]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:54 smithi184 ceph-mon[102844]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:54 smithi196 ceph-mon[76349]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:56 smithi184 ceph-mon[100622]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:56.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:56 smithi184 ceph-mon[102844]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:56 smithi196 ceph-mon[76349]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[102844]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[100622]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:58.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.800 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:58 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:58 smithi196 ceph-mon[76349]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:58 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:30:59.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:30:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:30:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:59.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:30:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[102844]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[100622]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:00 smithi196 ceph-mon[76349]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:02] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:31:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:02.173Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:02.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:02.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:02.750 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-mon[100622]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:02.751 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:02 smithi184 ceph-mon[102844]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:02 smithi196 ceph-mon[76349]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:04 smithi184 ceph-mon[102844]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:04 smithi184 ceph-mon[100622]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:04 smithi196 ceph-mon[76349]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:06.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:06 smithi184 ceph-mon[100622]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:06.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:06 smithi184 ceph-mon[102844]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:06 smithi196 ceph-mon[76349]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:08.169 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:31:08.702 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 10s ago 7h - - 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 8s ago 7h - - 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 10s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 8s ago 7h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 10s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 8s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 10s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 10s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 8s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 10s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 10s ago 7h 17.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:31:08.703 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 8s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 10s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 10s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 10s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 10s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 8s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 8s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 8s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 8s ago 7h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:31:08.704 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 8s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:31:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:08 smithi196 ceph-mon[76349]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:09.014 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:08 smithi184 ceph-mon[100622]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:09.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:08 smithi184 ceph-mon[102844]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:09.015 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:31:09.015 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:31:09.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:31:09.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:31:09.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:09.017 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:31:09.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:31:09.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:09.017 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:31:09.018 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:31:09.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:31:09.018 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:31:09.285 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:31:09.286 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:31:09.287 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:31:09.287 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:31:09.593 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:31:09.593 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:09.593 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:09.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:09 smithi196 ceph-mon[76349]: from='client.75213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:09.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:09 smithi196 ceph-mon[76349]: from='client.75219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:09.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:09 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/734555859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[100622]: from='client.75213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[100622]: from='client.75219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/734555859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[102844]: from='client.75213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[102844]: from='client.75219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:09 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/734555859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:10 smithi196 ceph-mon[76349]: from='client.85187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:10 smithi196 ceph-mon[76349]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:10 smithi196 ceph-mon[76349]: from='client.75237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:10 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/857224847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:11.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[100622]: from='client.85187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[100622]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[100622]: from='client.75237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/857224847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[102844]: from='client.85187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[102844]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[102844]: from='client.75237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:11.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:10 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/857224847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:12] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:31:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:12.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:12.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:12.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:12.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:12 smithi196 ceph-mon[76349]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:13.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-mon[100622]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:13.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:12 smithi184 ceph-mon[102844]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:14 smithi196 ceph-mon[76349]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:14 smithi184 ceph-mon[102844]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:15.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:14 smithi184 ceph-mon[100622]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:15.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:16 smithi196 ceph-mon[76349]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:16 smithi184 ceph-mon[100622]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:16 smithi184 ceph-mon[102844]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:18 smithi196 ceph-mon[76349]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:18 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[102844]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[100622]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:18 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:20 smithi196 ceph-mon[76349]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:20.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[102844]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:21.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[100622]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:21.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:22] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:31:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:22.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:22.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:22.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:22.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:22 smithi196 ceph-mon[76349]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:23.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-mon[100622]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:23.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:22 smithi184 ceph-mon[102844]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:23.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:23 smithi196 ceph-mon[76349]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:23.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:23 smithi184 ceph-mon[100622]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:23.926 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:23 smithi184 ceph-mon[102844]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:26.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:25 smithi184 ceph-mon[100622]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:26.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:25 smithi184 ceph-mon[102844]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:25 smithi196 ceph-mon[76349]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:27 smithi184 ceph-mon[100622]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:27 smithi184 ceph-mon[102844]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:27 smithi196 ceph-mon[76349]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:30.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:29 smithi196 ceph-mon[76349]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:30.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:30.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[100622]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:30.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[102844]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:32.178 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:31 smithi184 ceph-mon[100622]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:32.178 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:32] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:31:32.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:31 smithi184 ceph-mon[102844]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:32.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:31 smithi196 ceph-mon[76349]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:32.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:32.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:32.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:32.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:32.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:32.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:34.291 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:33 smithi184 ceph-mon[100622]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:34.291 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:33 smithi184 ceph-mon[102844]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:34.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:33 smithi196 ceph-mon[76349]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:36.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:35 smithi196 ceph-mon[76349]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:35 smithi184 ceph-mon[100622]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:36.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:35 smithi184 ceph-mon[102844]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:37 smithi184 ceph-mon[100622]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:37 smithi184 ceph-mon[102844]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:37 smithi196 ceph-mon[76349]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:39.888 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:31:40.423 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 2s ago 7h - - 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 0s ago 7h - - 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 2s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 0s ago 7h 95.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 2s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 0s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 2s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 2s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 2s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 2s ago 7h 17.0M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 0s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:31:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 2s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 2s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 2s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 2s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 0s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 0s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 0s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 0s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 0s ago 7h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:31:40.425 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 0s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:31:40.737 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:31:40.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[100622]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:40.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:40.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.737 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[102844]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.738 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:40 smithi196 ceph-mon[76349]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:41.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:31:41.002 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:31:41.002 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:31:41.002 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:31:41.003 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:31:41.310 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:31:41.311 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:41.311 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:41.311 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:41.311 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[100622]: from='client.85211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[100622]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[100622]: from='client.85223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2449985052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1127537795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[102844]: from='client.85211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[102844]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[102844]: from='client.85223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2449985052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:41 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1127537795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:41 smithi196 ceph-mon[76349]: from='client.85211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:41 smithi196 ceph-mon[76349]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:41 smithi196 ceph-mon[76349]: from='client.85223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2449985052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:41 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1127537795' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:42] "GET /metrics HTTP/1.1" 200 38942 "" "Prometheus/2.43.0" 2024-09-21T03:31:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:42.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:42.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:42.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-mon[100622]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-mon[100622]: from='client.75273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-mon[102844]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:42.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:42 smithi184 ceph-mon[102844]: from='client.75273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:42 smithi196 ceph-mon[76349]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:42 smithi196 ceph-mon[76349]: from='client.75273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:44.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:44 smithi184 ceph-mon[100622]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:44.730 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:44.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:44 smithi184 ceph-mon[102844]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:44.730 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:44 smithi196 ceph-mon[76349]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:46.731 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:46 smithi184 ceph-mon[100622]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:46.731 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:46 smithi184 ceph-mon[102844]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:46 smithi196 ceph-mon[76349]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:48.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:48 smithi184 ceph-mon[100622]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:48.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:48 smithi184 ceph-mon[102844]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:48 smithi196 ceph-mon[76349]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:50 smithi196 ceph-mon[76349]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:50 smithi184 ceph-mon[100622]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:50.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:50.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:50 smithi184 ceph-mon[102844]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:31:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:31:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:52.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:52.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:31:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:31:52.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:31:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-mon[100622]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:52.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:52 smithi184 ceph-mon[102844]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:52.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:52 smithi196 ceph-mon[76349]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:54 smithi184 ceph-mon[100622]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:54 smithi184 ceph-mon[102844]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:54 smithi196 ceph-mon[76349]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:56 smithi184 ceph-mon[100622]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:56 smithi184 ceph-mon[102844]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:56 smithi196 ceph-mon[76349]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:58 smithi184 ceph-mon[100622]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:58.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:58 smithi184 ceph-mon[102844]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:58 smithi196 ceph-mon[76349]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:31:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:31:59.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:31:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:00.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[102844]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[100622]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:00 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:00 smithi196 ceph-mon[76349]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:00 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:00.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:00 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:02] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:32:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:02.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:02.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:02.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:02.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-mon[100622]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:02.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:02 smithi184 ceph-mon[102844]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:02 smithi196 ceph-mon[76349]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:04.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:04 smithi184 ceph-mon[100622]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:04.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:04 smithi184 ceph-mon[102844]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:04 smithi196 ceph-mon[76349]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:06.789 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:06 smithi184 ceph-mon[100622]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:06.790 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:06 smithi184 ceph-mon[102844]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:06 smithi196 ceph-mon[76349]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:08.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:08 smithi184 ceph-mon[100622]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:08.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:08 smithi184 ceph-mon[102844]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:08.884 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:08 smithi196 ceph-mon[76349]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:10 smithi184 ceph-mon[100622]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:10.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:10 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:10 smithi184 ceph-mon[102844]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:10.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:10 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:10 smithi196 ceph-mon[76349]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:10.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:10 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:11.606 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:32:12.133 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:32:12.133 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 13s ago 7h - - 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 11s ago 7h - - 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 13s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 11s ago 7h 96.0M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 13s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 11s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 13s ago 7h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 13s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 11s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 13s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 13s ago 7h 18.2M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 11s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 13s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 13s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 13s ago 7h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 13s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:32:12.134 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 11s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 11s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 11s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 11s ago 7h 71.1M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 13s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 13s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:32:12.135 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 11s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:32:12.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:12] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:32:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:12.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:12.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:12.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:12.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:32:12.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:32:12.448 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:32:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:12 smithi196 ceph-mon[76349]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:12 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4224721911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:32:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:32:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:32:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:32:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:32:12.721 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:32:12.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-mon[100622]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.721 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4224721911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:12.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-mon[102844]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.721 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:12 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4224721911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:13.030 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:32:13.030 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:13.030 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:13.030 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:13.031 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:13 smithi196 ceph-mon[76349]: from='client.75285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:13 smithi196 ceph-mon[76349]: from='client.75291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:13 smithi196 ceph-mon[76349]: from='client.75297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:13.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:13 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/1569140418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[100622]: from='client.75285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[100622]: from='client.75291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[100622]: from='client.75297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/1569140418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[102844]: from='client.75285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[102844]: from='client.75291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[102844]: from='client.75297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:13 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/1569140418' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:14 smithi196 ceph-mon[76349]: from='client.85271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:14 smithi196 ceph-mon[76349]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:14.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[100622]: from='client.85271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[100622]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:15.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[102844]: from='client.85271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[102844]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:15.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:16.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:16 smithi196 ceph-mon[76349]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:17.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:16 smithi184 ceph-mon[100622]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:17.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:16 smithi184 ceph-mon[102844]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:18.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:18 smithi196 ceph-mon[76349]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:19.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:18 smithi184 ceph-mon[100622]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:19.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:18 smithi184 ceph-mon[102844]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:19 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:19 smithi196 ceph-mon[76349]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:19.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:19 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[100622]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[102844]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:20.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:19 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:21.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:22] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:32:22.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:22.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:22.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:22.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:22.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:22 smithi196 ceph-mon[76349]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:22.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-mon[102844]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:22.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:22 smithi184 ceph-mon[100622]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:24.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:24 smithi196 ceph-mon[76349]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:24.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:24 smithi184 ceph-mon[102844]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:24.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:24 smithi184 ceph-mon[100622]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:26.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:26 smithi196 ceph-mon[76349]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:26.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:26 smithi184 ceph-mon[100622]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:26.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:26 smithi184 ceph-mon[102844]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:28.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:28 smithi196 ceph-mon[76349]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:28.713 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:28 smithi184 ceph-mon[100622]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:28.714 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:28 smithi184 ceph-mon[102844]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:29.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:29.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:29.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:30 smithi196 ceph-mon[76349]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:30.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:30 smithi184 ceph-mon[102844]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:30.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:30.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:30 smithi184 ceph-mon[100622]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:30.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:32.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:32] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:32:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:32.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:32.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:32.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:32.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:32 smithi196 ceph-mon[76349]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-mon[100622]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:32.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:32 smithi184 ceph-mon[102844]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:34.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:34 smithi196 ceph-mon[76349]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:34 smithi184 ceph-mon[100622]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:34 smithi184 ceph-mon[102844]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:36 smithi196 ceph-mon[76349]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:36.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:36 smithi184 ceph-mon[100622]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:36 smithi184 ceph-mon[102844]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:38.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:38 smithi196 ceph-mon[76349]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:38.702 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:38 smithi184 ceph-mon[100622]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:38.702 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:38 smithi184 ceph-mon[102844]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[102844]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[100622]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:40.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:39 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:39 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:39 smithi196 ceph-mon[76349]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:40.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:39 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:41.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:41.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:32:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:42] "GET /metrics HTTP/1.1" 200 38943 "" "Prometheus/2.43.0" 2024-09-21T03:32:42.298 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:42.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:42.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:42.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:42.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:42.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-mon[100622]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:42.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:42 smithi184 ceph-mon[102844]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:42.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:42 smithi196 ceph-mon[76349]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:43.326 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:32:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:32:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 5s ago 7h - - 2024-09-21T03:32:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 3s ago 7h - - 2024-09-21T03:32:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 5s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:32:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 3s ago 7h 94.2M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 5s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 3s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 5s ago 7h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 5s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 3s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 5s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 5s ago 7h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 3s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 5s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 5s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 5s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 5s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 3s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 3s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 3s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:32:43.853 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 3s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:32:43.854 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 3s ago 7h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:32:43.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:32:43.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:32:43.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:32:43.854 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 3s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:32:44.161 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:32:44.161 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:32:44.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:32:44.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:32:44.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:32:44.432 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:32:44.432 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:32:44.432 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:32:44.432 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:32:44.432 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:32:44.433 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:32:44.746 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:32:44.746 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:44.746 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:44.746 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[100622]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[100622]: from='client.85283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[100622]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3337798596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[102844]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[102844]: from='client.85283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[102844]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3337798596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:44.747 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:44 smithi196 ceph-mon[76349]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:44 smithi196 ceph-mon[76349]: from='client.85283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:44 smithi196 ceph-mon[76349]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:44 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3337798596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:44.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[100622]: from='client.75333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[100622]: from='client.75345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3062831310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[102844]: from='client.75333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[102844]: from='client.75345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:45 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3062831310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:45 smithi196 ceph-mon[76349]: from='client.75333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:45 smithi196 ceph-mon[76349]: from='client.75345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:45.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:45 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3062831310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:46.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:46 smithi184 ceph-mon[100622]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:46.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:46 smithi184 ceph-mon[102844]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:46.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:46 smithi196 ceph-mon[76349]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:48.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:48 smithi184 ceph-mon[100622]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:48 smithi184 ceph-mon[102844]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:48 smithi196 ceph-mon[76349]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:50 smithi184 ceph-mon[102844]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:50.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:50 smithi184 ceph-mon[100622]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:50.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:50 smithi196 ceph-mon[76349]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:50.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:52.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:32:52] "GET /metrics HTTP/1.1" 200 38948 "" "Prometheus/2.43.0" 2024-09-21T03:32:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:52.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:52.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:32:52.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:32:52.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:32:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:52 smithi196 ceph-mon[76349]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:52.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-mon[100622]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:52.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:52 smithi184 ceph-mon[102844]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:54.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:54 smithi184 ceph-mon[100622]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:54.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:54 smithi184 ceph-mon[102844]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:54.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:54 smithi196 ceph-mon[76349]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:56.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:56 smithi184 ceph-mon[100622]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:56.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:56 smithi184 ceph-mon[102844]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:56.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:56 smithi196 ceph-mon[76349]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:58.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:58 smithi184 ceph-mon[100622]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:58.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:58 smithi184 ceph-mon[102844]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:58.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:58 smithi196 ceph-mon[76349]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:32:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:32:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:01.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.003 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:02] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:33:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:02.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:02.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:02.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:02.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:02.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:02 smithi196 ceph-mon[76349]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:03.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-mon[100622]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:03.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:02 smithi184 ceph-mon[102844]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:04.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:04 smithi196 ceph-mon[76349]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:05.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:04 smithi184 ceph-mon[100622]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:05.048 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:04 smithi184 ceph-mon[102844]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:06.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:06 smithi196 ceph-mon[76349]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:07.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:06 smithi184 ceph-mon[100622]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:07.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:06 smithi184 ceph-mon[102844]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:08.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:08 smithi196 ceph-mon[76349]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:08.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:08 smithi184 ceph-mon[100622]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:08.921 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:08 smithi184 ceph-mon[102844]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:09.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:09 smithi184 ceph-mon[100622]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:09.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:09 smithi184 ceph-mon[102844]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:09.799 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:09 smithi196 ceph-mon[76349]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:10.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:12.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:11 smithi196 ceph-mon[76349]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:11 smithi184 ceph-mon[100622]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:11 smithi184 ceph-mon[102844]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:12] "GET /metrics HTTP/1.1" 200 38944 "" "Prometheus/2.43.0" 2024-09-21T03:33:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:12.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:12.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:12.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:12.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:14.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:13 smithi196 ceph-mon[76349]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:13 smithi184 ceph-mon[100622]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:13 smithi184 ceph-mon[102844]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:15.044 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:33:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:15.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 16s ago 7h - - 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 14s ago 7h - - 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 16s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 14s ago 7h 95.8M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 16s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 14s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 16s ago 7h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:33:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 16s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 14s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 16s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 16s ago 7h 17.9M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 14s ago 7h 17.5M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 16s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 16s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 16s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 14s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 14s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 14s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 14s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 14s ago 7h 71.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:33:15.575 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:33:15.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:33:15.576 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 14s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:33:15.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:15.886 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:33:15.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:33:15.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:33:15.886 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:33:16.155 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:33:16.155 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:33:16.155 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:33:16.155 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:33:16.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[100622]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[100622]: from='client.85319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[100622]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[100622]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/470600653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[102844]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[102844]: from='client.85319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[102844]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.157 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[102844]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.158 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:15 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/470600653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:15 smithi196 ceph-mon[76349]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:15 smithi196 ceph-mon[76349]: from='client.85319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:15 smithi196 ceph-mon[76349]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:15 smithi196 ceph-mon[76349]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:16.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:15 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/470600653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:16.465 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:33:16.465 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:16.465 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:16.466 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:16.466 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:16 smithi184 ceph-mon[100622]: from='client.85343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:17.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:16 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/4056839822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:16 smithi184 ceph-mon[102844]: from='client.85343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:17.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:16 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/4056839822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:16 smithi196 ceph-mon[76349]: from='client.85343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:17.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:16 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/4056839822' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:18.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:17 smithi184 ceph-mon[100622]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:17 smithi184 ceph-mon[102844]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:17 smithi196 ceph-mon[76349]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[100622]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[102844]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:20 smithi196 ceph-mon[76349]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:22.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[102844]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.005 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:21 smithi184 ceph-mon[100622]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:21 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:22.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:21 smithi196 ceph-mon[76349]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:22] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:33:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:22.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:22.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:22.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:22.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:24.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:23 smithi184 ceph-mon[100622]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:24.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:23 smithi184 ceph-mon[102844]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:23 smithi196 ceph-mon[76349]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:26.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:25 smithi184 ceph-mon[100622]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:26.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:25 smithi184 ceph-mon[102844]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:25 smithi196 ceph-mon[76349]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:28.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:27 smithi184 ceph-mon[100622]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:28.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:27 smithi184 ceph-mon[102844]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:27 smithi196 ceph-mon[76349]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[100622]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[102844]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:30.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:29 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:29 smithi196 ceph-mon[76349]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:29 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:29 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:32.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:31 smithi196 ceph-mon[76349]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:32.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:31 smithi184 ceph-mon[100622]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:32.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:31 smithi184 ceph-mon[102844]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:32.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:32] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:33:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:32.183Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:32.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:32.299 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:32.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:34.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:33 smithi184 ceph-mon[100622]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:34.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:33 smithi184 ceph-mon[102844]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:34.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:33 smithi196 ceph-mon[76349]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:36.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:35 smithi184 ceph-mon[100622]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:36.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:35 smithi184 ceph-mon[102844]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:36.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:35 smithi196 ceph-mon[76349]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:38.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:37 smithi184 ceph-mon[100622]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:38.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:37 smithi184 ceph-mon[102844]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:38.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:37 smithi196 ceph-mon[76349]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[100622]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[102844]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:40 smithi196 ceph-mon[76349]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:40.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:41 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:41 smithi196 ceph-mon[76349]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:42] "GET /metrics HTTP/1.1" 200 38947 "" "Prometheus/2.43.0" 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[100622]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:33:42.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:41 smithi184 ceph-mon[102844]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:42.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:42.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:42.548 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:42.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:44.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:43 smithi184 ceph-mon[100622]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:44.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:43 smithi184 ceph-mon[102844]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:44.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:43 smithi196 ceph-mon[76349]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:45.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:44 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:45.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:44 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:45.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:44 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:46.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:45 smithi184 ceph-mon[100622]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:46.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:45 smithi184 ceph-mon[102844]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:46.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:45 smithi196 ceph-mon[76349]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:46.761 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 8s ago 7h - - 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 6s ago 7h - - 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 8s ago 7h 28.1M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 6s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 8s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:33:47.289 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 6s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 8s ago 7h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 8s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 6s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 8s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 8s ago 7h 18.3M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 6s ago 7h 17.6M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 8s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 8s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 8s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 8s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 6s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 6s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:33:47.290 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 6s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 6s ago 7h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:33:47.291 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 6s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:33:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:33:47.599 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:33:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:33:47.869 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:33:48.179 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:33:48.180 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:48.180 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:48.180 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:48.180 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:48.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[102844]: from='client.85355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[102844]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:48.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[102844]: from='client.85361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[102844]: from='client.85367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2382633588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[100622]: from='client.85355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[100622]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[100622]: from='client.85361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[100622]: from='client.85367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.181 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:47 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2382633588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:48.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:47 smithi196 ceph-mon[76349]: from='client.85355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:47 smithi196 ceph-mon[76349]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:47 smithi196 ceph-mon[76349]: from='client.85361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:47 smithi196 ceph-mon[76349]: from='client.85367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:48.391 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:47 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2382633588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:49.246 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:48 smithi184 ceph-mon[100622]: from='client.85379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:49.246 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:48 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/66711518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:49.246 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:48 smithi184 ceph-mon[102844]: from='client.85379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:49.246 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:48 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/66711518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:48 smithi196 ceph-mon[76349]: from='client.85379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:49.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:48 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/66711518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:50.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:49 smithi184 ceph-mon[100622]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:50.243 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:49 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:50.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:49 smithi184 ceph-mon[102844]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:50.243 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:49 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:49 smithi196 ceph-mon[76349]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:50.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:49 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:52.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:51 smithi184 ceph-mon[100622]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:52.049 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:33:52] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:33:52.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:51 smithi184 ceph-mon[102844]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:52.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:51 smithi196 ceph-mon[76349]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:52.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:52.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:52.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:52.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:33:52.549 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:33:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:33:52.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:33:54.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:53 smithi196 ceph-mon[76349]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:54.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:53 smithi184 ceph-mon[100622]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:54.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:53 smithi184 ceph-mon[102844]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:56.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:55 smithi184 ceph-mon[100622]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:56.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:55 smithi184 ceph-mon[102844]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:56.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:55 smithi196 ceph-mon[76349]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:58.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:57 smithi184 ceph-mon[100622]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:58.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:57 smithi184 ceph-mon[102844]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:58.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:57 smithi196 ceph-mon[76349]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:00.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:33:59 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:00.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:33:59 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:02.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:02] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[102844]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[100622]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.300 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:01 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:02.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:02.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:02.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:02.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:02 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:02.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:01 smithi196 ceph-mon[76349]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:02.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:01 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:04.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:03 smithi184 ceph-mon[102844]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:04.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:03 smithi184 ceph-mon[100622]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:04.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:03 smithi196 ceph-mon[76349]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:06.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:05 smithi184 ceph-mon[102844]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:06.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:05 smithi184 ceph-mon[100622]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:06.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:05 smithi196 ceph-mon[76349]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:08.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:07 smithi184 ceph-mon[102844]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:34:08.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:07 smithi184 ceph-mon[100622]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:34:08.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:07 smithi196 ceph-mon[76349]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:34:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:09 smithi184 ceph-mon[102844]: pgmap v13825: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:34:10.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:09 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:09 smithi184 ceph-mon[100622]: pgmap v13825: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:34:10.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:09 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:09 smithi196 ceph-mon[76349]: pgmap v13825: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:34:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:09 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:12.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:11 smithi184 ceph-mon[100622]: pgmap v13826: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:12.299 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:12] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:34:12.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:11 smithi184 ceph-mon[102844]: pgmap v13826: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:12.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:12.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:12.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:12 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:12.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:12.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:11 smithi196 ceph-mon[76349]: pgmap v13826: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:14.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:13 smithi184 ceph-mon[100622]: pgmap v13827: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:14.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:13 smithi184 ceph-mon[102844]: pgmap v13827: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:14.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:13 smithi196 ceph-mon[76349]: pgmap v13827: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:34:15.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:14 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:15.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:14 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:15.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:14 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:16.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:15 smithi196 ceph-mon[76349]: pgmap v13828: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:34:16.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:15 smithi184 ceph-mon[100622]: pgmap v13828: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:34:16.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:15 smithi184 ceph-mon[102844]: pgmap v13828: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:34:18.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:17 smithi184 ceph-mon[100622]: pgmap v13829: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:34:18.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:17 smithi184 ceph-mon[102844]: pgmap v13829: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:34:18.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:17 smithi196 ceph-mon[76349]: pgmap v13829: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:34:18.481 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:34:19.012 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 19s ago 7h - - 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 18s ago 7h - - 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 19s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 18s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 19s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 18s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 19s ago 7h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 19s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 18s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 19s ago 7h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 19s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:34:19.013 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 18s ago 7h 17.2M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 19s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 19s ago 7h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 19s ago 7h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 18s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 18s ago 7h 98.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 18s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 18s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 18s ago 7h 71.5M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:34:19.014 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 18s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:34:19.298 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:18 smithi184 ceph-mon[102844]: from='client.75429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:19.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:18 smithi184 ceph-mon[100622]: from='client.75429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:34:19.333 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:34:19.334 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:34:19.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:19 smithi196 ceph-mon[76349]: from='client.75429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:19.656 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:34:19.657 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:34:19.658 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:34:19.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:34:19.975 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:34:19.975 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:19.975 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:19.975 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:19.976 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='client.85397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: pgmap v13830: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='client.85403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2138412733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/775420860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='client.85397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: pgmap v13830: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='client.85403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2138412733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:20.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:20 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/775420860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='client.85397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: pgmap v13830: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='client.85403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2138412733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:20.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:20 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/775420860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:21.298 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:21 smithi184 ceph-mon[100622]: from='client.85415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:21.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:21 smithi184 ceph-mon[102844]: from='client.85415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:21.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:21 smithi196 ceph-mon[76349]: from='client.85415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:22.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:22] "GET /metrics HTTP/1.1" 200 38945 "" "Prometheus/2.43.0" 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[100622]: pgmap v13831: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[102844]: pgmap v13831: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:22.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:22.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:22.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:22.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:22 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:22.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:22 smithi196 ceph-mon[76349]: pgmap v13831: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T03:34:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:22.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:22 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:24.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:24 smithi196 ceph-mon[76349]: pgmap v13832: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:34:24.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:24 smithi184 ceph-mon[100622]: pgmap v13832: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:34:24.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:24 smithi184 ceph-mon[102844]: pgmap v13832: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T03:34:26.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:26 smithi196 ceph-mon[76349]: pgmap v13833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:26.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:26 smithi184 ceph-mon[100622]: pgmap v13833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:26.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:26 smithi184 ceph-mon[102844]: pgmap v13833: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:28.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:28 smithi196 ceph-mon[76349]: pgmap v13834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:28.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:28 smithi184 ceph-mon[100622]: pgmap v13834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:28.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:28 smithi184 ceph-mon[102844]: pgmap v13834: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:34:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:30 smithi196 ceph-mon[76349]: pgmap v13835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:34:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:30 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:30.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:30 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[100622]: pgmap v13835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[102844]: pgmap v13835: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:30.495 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:30 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:32.190 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:32] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:34:32.530 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:32.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:32.530 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:32.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:32.530 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:32.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:32.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-mon[100622]: pgmap v13836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:34:32.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:32 smithi184 ceph-mon[102844]: pgmap v13836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:34:32.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:32 smithi196 ceph-mon[76349]: pgmap v13836: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T03:34:34.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:34 smithi184 ceph-mon[100622]: pgmap v13837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:34:34.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:34 smithi184 ceph-mon[102844]: pgmap v13837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:34:34.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:34 smithi196 ceph-mon[76349]: pgmap v13837: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:34:36.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:36 smithi196 ceph-mon[76349]: pgmap v13838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:34:36.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:36 smithi184 ceph-mon[102844]: pgmap v13838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:34:36.799 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:36 smithi184 ceph-mon[100622]: pgmap v13838: 161 pgs: 161 active+clean; 585 KiB data, 427 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:34:38.798 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:38 smithi184 ceph-mon[100622]: pgmap v13839: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:38.798 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:38 smithi184 ceph-mon[102844]: pgmap v13839: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:38.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:38 smithi196 ceph-mon[76349]: pgmap v13839: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:40 smithi196 ceph-mon[76349]: pgmap v13840: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:40 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:40.890 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:40 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.048 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[100622]: pgmap v13840: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[102844]: pgmap v13840: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:41.049 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:40 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.298 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:42] "GET /metrics HTTP/1.1" 200 38946 "" "Prometheus/2.43.0" 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[100622]: pgmap v13841: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[102844]: pgmap v13841: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:34:42.299 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.300 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:42.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:42.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:42.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:42.300 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:42 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:42.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:42 smithi196 ceph-mon[76349]: pgmap v13841: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:34:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:42.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:42 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' 2024-09-21T03:34:44.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:44 smithi184 ceph-mon[102844]: pgmap v13842: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:34:44.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:44 smithi184 ceph-mon[100622]: pgmap v13842: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:34:44.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:44 smithi196 ceph-mon[76349]: pgmap v13842: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T03:34:45.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:45 smithi184 ceph-mon[100622]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:45.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:45 smithi184 ceph-mon[102844]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:45.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:45 smithi196 ceph-mon[76349]: from='mgr.34107 172.21.15.184:0/687885594' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:46.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:46 smithi184 ceph-mon[100622]: pgmap v13843: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:34:46.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:46 smithi184 ceph-mon[102844]: pgmap v13843: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:34:46.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:46 smithi196 ceph-mon[76349]: pgmap v13843: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T03:34:48.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:48 smithi184 ceph-mon[100622]: pgmap v13844: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T03:34:48.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:48 smithi184 ceph-mon[102844]: pgmap v13844: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T03:34:48.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:48 smithi196 ceph-mon[76349]: pgmap v13844: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-21T03:34:50.272 INFO:teuthology.orchestra.run.smithi184.stdout:true 2024-09-21T03:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:50 smithi184 ceph-mon[100622]: pgmap v13845: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T03:34:50.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:50 smithi184 ceph-mon[100622]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:50.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:50 smithi184 ceph-mon[102844]: pgmap v13845: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T03:34:50.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:50 smithi184 ceph-mon[102844]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:50 smithi196 ceph-mon[76349]: pgmap v13845: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-21T03:34:50.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:50 smithi196 ceph-mon[76349]: from='client.25192 -' entity='client.iscsi.foo.smithi184.evkrpb' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 11s ago 7h - - 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi196 smithi196 running 9s ago 7h - - 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (7h) 11s ago 7h 28.2M - 0.25.0 c8568f914cd2 3d35fb41de6e 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi196 *:3000 running (7h) 9s ago 7h 96.3M - 9.4.7 954c08fa6188 16f45b96cac7 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.foo.smithi184.evkrpb smithi184 *:5000 running (7h) 11s ago 7h 62.9M - 3.8 4404209dfb71 8b5e4f4232e0 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi196 *:9283,8765 running (7h) 9s ago 7h 460M - 19.3.0-5127-ge8baca1e 82f5f45939a4 ad8e16e70e79 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (7h) 11s ago 7h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 478ddee08a13 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (7h) 11s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 20c10837ba0d 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi196 running (7h) 9s ago 7h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 f9e3f0fb6c5e 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi184 running (7h) 11s ago 7h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 b6535806fb3c 2024-09-21T03:34:50.802 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 running (7h) 11s ago 7h 17.4M - 1.5.0 0da6a335fe13 cfe7897b375d 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b smithi196 *:9100 running (7h) 9s ago 7h 17.3M - 1.5.0 0da6a335fe13 edb7c108b4f8 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (7h) 11s ago 7h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 8a6cac5c11f0 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi184 running (7h) 11s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 1eab7b64494b 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi184 running (7h) 11s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 04c43acf3cec 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3 smithi184 running (7h) 11s ago 7h 114M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b06017b98baf 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4 smithi196 running (7h) 9s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 268ec6ab1596 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5 smithi196 running (7h) 9s ago 7h 98.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ba91460ce77d 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6 smithi196 running (7h) 9s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 82dcd5713f0f 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7 smithi196 running (7h) 9s ago 7h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 3e53ce58d0b3 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi196 *:9095 running (7h) 9s ago 7h 72.2M - 2.43.0 a07b618ecd1d 9515b5acbfdd 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi184.ctohee smithi184 *:8000 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 e2698c014b3b 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.smithi196.zyivrs smithi196 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 439e8c28f30d 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi184.fllvrf smithi184 *:80 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 089bdb87426f 2024-09-21T03:34:50.803 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.smpl.smithi196.vsmvti smithi196 *:80 running (7h) 9s ago 7h 91.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 87991c436cb7 2024-09-21T03:34:51.124 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:34:51.124 INFO:teuthology.orchestra.run.smithi184.stdout: "mon": { 2024-09-21T03:34:51.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:34:51.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:51.124 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr": { 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "osd": { 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw": { 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "overall": { 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-21T03:34:51.125 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[100622]: from='client.85427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[100622]: from='client.85433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/2584735849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[102844]: from='client.85427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[102844]: from='client.85433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.312 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:51 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/2584735849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "in_progress": true, 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "services_complete": [ 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "rgw", 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "mgr", 2024-09-21T03:34:51.418 INFO:teuthology.orchestra.run.smithi184.stdout: "mon", 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout: "osd" 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout: "is_paused": false 2024-09-21T03:34:51.419 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-21T03:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:51 smithi196 ceph-mon[76349]: from='client.85427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:51 smithi196 ceph-mon[76349]: from='client.85433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:51.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:51 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/2584735849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:51.734 INFO:teuthology.orchestra.run.smithi184.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:34:51.734 INFO:teuthology.orchestra.run.smithi184.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:51.734 INFO:teuthology.orchestra.run.smithi184.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:52.254 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[100622]: from='client.85439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.254 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-mgr-y[86527]: ::ffff:172.21.15.196 - - [21/Sep/2024:03:34:52] "GET /metrics HTTP/1.1" 200 38949 "" "Prometheus/2.43.0" 2024-09-21T03:34:52.254 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:52.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:52.254 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:52.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi184.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.184:8443: connect: connection refused" 2024-09-21T03:34:52.254 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-4b598dd2-7788-11ef-baf5-efdc52797490-alertmanager-a[59203]: ts=2024-09-21T03:34:52.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-09-21T03:34:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[100622]: pgmap v13846: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:34:52.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[100622]: from='client.85451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[100622]: from='client.? 172.21.15.184:0/3952744890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[102844]: from='client.85439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[102844]: pgmap v13846: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:34:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[102844]: from='client.85451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:52 smithi184 ceph-mon[102844]: from='client.? 172.21.15.184:0/3952744890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:52 smithi196 ceph-mon[76349]: from='client.85439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:52 smithi196 ceph-mon[76349]: pgmap v13846: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:34:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:52 smithi196 ceph-mon[76349]: from='client.85451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:52.640 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:52 smithi196 ceph-mon[76349]: from='client.? 172.21.15.184:0/3952744890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:54.514 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:54 smithi196 ceph-mon[76349]: pgmap v13847: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:34:54.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:54 smithi184 ceph-mon[100622]: pgmap v13847: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:34:54.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:54 smithi184 ceph-mon[102844]: pgmap v13847: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:34:56.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 21 03:34:56 smithi196 ceph-mon[76349]: pgmap v13848: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T03:34:56.548 INFO:journalctl@ceph.mon.a.smithi184.stdout:Sep 21 03:34:56 smithi184 ceph-mon[100622]: pgmap v13848: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T03:34:56.549 INFO:journalctl@ceph.mon.c.smithi184.stdout:Sep 21 03:34:56 smithi184 ceph-mon[102844]: pgmap v13848: 161 pgs: 161 active+clean; 585 KiB data, 449 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T03:34:57.714 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-21T03:34:57.714 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2024-09-21T03:34:57.715 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2024-09-21T03:34:57.715 DEBUG:teuthology.exit:Finished running handlers